Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training

Abstract

Distributed training of deep nets is an important technique to address some of the present day computing challenges like memory consumption and computational demands. Classical distributed approaches, synchronous or asynchronous, are based on the parameter server architecture, i.e., worker nodes compute gradients which are communicated to the parameter server while updated parameters are returned. Recently, distributed training with AllReduce operations gained popularity as well. While many of those operations seem appealing, little is reported about wall-clock training time improvements. In this paper, we carefully analyze the AllReduce based setup, propose timing models which include network latency, bandwidth, cluster size and compute time, and demonstrate that a pipelined training with a width of two combines the best of both synchronous and asynchronous training. Specifically, for a setup consisting of a four-node GPU cluster we show wall-clock time training improvements of up to 5.4x compared to conventional approaches.

arXiv commentAccepted at NeurIPS 2018

References in corpus (8)

• Playing Atari with Deep Reinforcement Learning
• Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• FractalNet: Ultra-Deep Neural Networks without Residuals
• AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training
• DeepSpark: A Spark-Based Distributed Deep Learning Framework for Commodity Clusters

Cited by in corpus (18)

• Central Server Free Federated Learning over Single-sided Trust Social Networks
• Turbo-Aggregate: Breaking the Quadratic Aggregation Barrier in Secure Federated Learning
• TornadoAggregate: Accurate and Scalable Federated Learning via the Ring-Based Architecture
• Data Movement Is All You Need: A Case Study on Optimizing Transformers
• $\text{DeepSqueeze}$: Decentralization Meets Error-Compensated Compression
• Layer-wise Adaptive Gradient Sparsification for Distributed Deep Learning with Convergence Guarantees
• LayerPipe: Accelerating Deep Neural Network Training by Intra-Layer and Inter-Layer Gradient Pipelining and Multiprocessor Scheduling
• The Case for Strong Scaling in Deep Learning: Training Large 3D CNNs with Hybrid Parallelism
• Heterogeneity-Aware Asynchronous Decentralized Training
• DBS: Dynamic Batch Size For Distributed Deep Neural Network Training
• Predicting Weather Uncertainty with Deep Convnets
• Intermittent Pulling with Local Compensation for Communication-Efficient Federated Learning
• APMSqueeze: A Communication Efficient Adam-Preconditioned Momentum SGD Algorithm
• MixML: A Unified Analysis of Weakly Consistent Parallel Learning
• Faster Distributed Deep Net Training: Computation and Communication Decoupled Stochastic Gradient Descent
• 1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed
• CD-SGD: Distributed Stochastic Gradient Descent with Compression and Delay Compensation
• Shuffle-Exchange Brings Faster: Reduce the Idle Time During Communication for Decentralized Neural Network Training