SparCML: High-Performance Sparse Communication for Machine Learning

Abstract

Applying machine learning techniques to the quickly growing data in science and industry requires highly-scalable algorithms. Large datasets are most commonly processed "data parallel" distributed across many nodes. Each node's contribution to the overall gradient is summed using a global allreduce. This allreduce is the single communication and thus scalability bottleneck for most machine learning workloads. We observe that frequently, many gradient values are (close to) zero, leading to sparse of sparsifyable communications. To exploit this insight, we analyze, design, and implement a set of communication-efficient protocols for sparse input data, in conjunction with efficient machine learning algorithms which can leverage these primitives. Our communication protocols generalize standard collective operations, by allowing processes to contribute arbitrary sparse input data vectors. Our generic communication library, SparCML, extends MPI to support additional features, such as non-blocking (asynchronous) operations and low-precision data representations. As such, SparCML and its techniques will form the basis of future highly-scalable machine learning frameworks.

References in corpus (14)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Large Batch Training of Convolutional Networks
• Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization
• Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting
• YellowFin and the Art of Momentum Tuning
• Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to train
• The Effect of Network Width on the Performance of Large-batch Training

Cited by in corpus (7)

• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• Taming Unbalanced Training Workloads in Deep Learning with Partial Collective Operations
• RedSync : Reducing Synchronization Traffic for Distributed Deep Learning
• Sparse Communication for Training Deep Networks
• The Convergence of Sparsified Gradient Methods
• Sparsity in Deep Neural Networks - An Empirical Investigation with TensorQuant
• Gradient Sparification for Asynchronous Distributed Training