Communication-Efficient Distributed Deep Learning: A Comprehensive Survey

Abstract

Distributed deep learning (DL) has become prevalent in recent years to reduce training time by leveraging multiple computing devices (e.g., GPUs/TPUs) due to larger models and datasets. However, system scalability is limited by communication becoming the performance bottleneck. Addressing this communication issue has become a prominent research topic. In this paper, we provide a comprehensive survey of the communication-efficient distributed training algorithms, focusing on both system-level and algorithmic-level optimizations. We first propose a taxonomy of data-parallel distributed training algorithms that incorporates four primary dimensions: communication synchronization, system architectures, compression techniques, and parallelism of communication and computing tasks. We then investigate state-of-the-art studies that address problems in these four dimensions. We also compare the convergence rates of different algorithms to understand their convergence speed. Additionally, we conduct extensive experiments to empirically compare the convergence performance of various mainstream distributed training algorithms. Based on our system-level communication cost analysis, theoretical and experimental convergence speed comparison, we provide readers with an understanding of which algorithms are more efficient under specific distributed environments. Our research also extrapolates potential directions for further optimizations.

References in corpus (61)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• cuDNN: Efficient Primitives for Deep Learning
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• One weird trick for parallelizing convolutional neural networks
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Revisiting Distributed Synchronous SGD
• D-ADMM: A Communication-Efficient Distributed Algorithm For Separable Optimization
• Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• Device Placement Optimization with Reinforcement Learning
• Sparsified SGD with Memory
• Don't Use Large Mini-Batches, Use Local SGD
• A Survey on Methods and Theories of Quantized Neural Networks
• ATOMO: Communication-efficient Learning via Atomic Sparsification
• Optimal algorithms for smooth and strongly convex distributed optimization in networks
• Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication
• Communication Compression for Decentralized Training
• D${}^2$: Decentralized Training over Decentralized Data
• A Universal Catalyst for First-Order Optimization
• Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
• Error Feedback Fixes SignSGD and other Gradient Compression Schemes
• Measuring the Effects of Data Parallelism on Neural Network Training
• On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization
• Distributed Mean Estimation with Limited Communication
• Stochastic Gradient Push for Distributed Deep Learning
• Taming the Wild: A Unified Analysis of Hogwild!-Style Algorithms
• meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting
• Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization
• Distributed Learning with Compressed Gradient Differences
• A Survey of Deep Learning Techniques for Neural Machine Translation
• PipeDream: Fast and Efficient Pipeline Parallel DNN Training
• GossipGraD: Scalable Deep Learning using Gossip Communication based Asynchronous Gradient Descent
• AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training
• Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash
• Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedback
• Understanding Top-k Sparsification in Distributed Deep Learning
• Natural Compression for Distributed Deep Learning
• Communication-Efficient Distributed Learning via Lazily Aggregated Quantized Gradients
• How to scale distributed deep learning?
• Variance-based Gradient Compression for Efficient Distributed Deep Learning
• Priority-based Parameter Propagation for Distributed DNN Training
• Asynchronous Coordinate Descent under More Realistic Assumptions
• COLA: Decentralized Linear Learning
• PASSCoDe: Parallel ASynchronous Stochastic dual Co-ordinate Descent
• Global Convergence of Block Coordinate Descent in Deep Learning
• On the Computation and Communication Complexity of Parallel SGD with Dynamic Batch Sizes for Stochastic Non-Convex Optimization
• TicTac: Accelerating Distributed Deep Learning with Communication Scheduling
• Blink: Fast and Generic Collectives for Distributed ML
• A distributed block coordinate descent method for training $l_1$ regularized linear classifiers
• A Proximal Block Coordinate Descent Algorithm for Deep Neural Network Training
• Communication-Efficient Decentralized Learning with Sparsification and Adaptive Peer Selection
• On Data Dependence in Distributed Stochastic Optimization
• Taming Convergence for Asynchronous Stochastic Gradient Descent with Unbounded Delay in Non-Convex Learning
• Communication Contention Aware Scheduling of Multiple Deep Learning Training Jobs
• Efficient Use of Limited-Memory Accelerators for Linear Learning on Heterogeneous Systems

Cited by in corpus (16)

• Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines
• Privacy-preserving Artificial Intelligence Techniques in Biomedicine
• An Overview of Federated Learning at the Edge and Distributed Ledger Technologies for Robotic and Autonomous Systems
• The Evolution of Distributed Systems for Graph Neural Networks and their Origin in Graph Processing and Deep Learning: A Survey
• On-Device Machine Learning: An Algorithms and Learning Theory Perspective
• Towards Efficient Synchronous Federated Training: A Survey on System Optimization Strategies
• On the Utility of Gradient Compression in Distributed Training Systems
• Local SGD With a Communication Overhead Depending Only on the Number of Workers
• Straggler-Resilient Distributed Machine Learning with Dynamic Backup Workers
• Large-Scale Deep Learning Optimizations: A Comprehensive Survey
• Communication-efficient SGD: From Local SGD to One-Shot Averaging
• More Industry-friendly: Federated Learning with High Efficient Design
• ResIST: Layer-Wise Decomposition of ResNets for Distributed Training
• Towards a Standardized Representation for Deep Learning Collective Algorithms
• Communication-Efficient Distributed Learning via Sparse and Adaptive Stochastic Gradient
• Privacy-Preserving Serverless Edge Learning with Decentralized Small Data