Communication optimization strategies for distributed deep neural network training: A survey

Abstract

Recent trends in high-performance computing and deep learning have led to the proliferation of studies on large-scale deep neural network training. However, the frequent communication requirements among computation nodes drastically slows the overall training speeds, which causes bottlenecks in distributed training, particularly in clusters with limited network bandwidths. To mitigate the drawbacks of distributed communications, researchers have proposed various optimization strategies. In this paper, we provide a comprehensive survey of communication strategies from both an algorithm viewpoint and a computer network perspective. Algorithm optimizations focus on reducing the communication volumes used in distributed training, while network optimizations focus on accelerating the communications between distributed devices. At the algorithm level, we describe how to reduce the number of communication rounds and transmitted bits per round. In addition, we elucidate how to overlap computation and communication. At the network level, we discuss the effects caused by network infrastructures, including logical communication schemes and network protocols. Finally, we extrapolate the potential future challenges and new research directions to accelerate communications for distributed deep neural network training.

References in corpus (45)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• TensorFlow: A system for large-scale machine learning
• Language Models are Few-Shot Learners
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• One weird trick for parallelizing convolutional neural networks
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Don't Decay the Learning Rate, Increase the Batch Size
• Sparse Communication for Distributed Gradient Descent
• Optimal Distributed Online Prediction using Mini-Batches
• Horovod: fast and easy distributed deep learning in TensorFlow
• Large Batch Training of Convolutional Networks
• ResNeSt: Split-Attention Networks
• Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes
• Sparsified SGD with Memory
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• Gradient Sparsification for Communication-Efficient Distributed Optimization
• ATOMO: Communication-efficient Learning via Atomic Sparsification
• Error Feedback Fixes SignSGD and other Gradient Compression Schemes
• PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization
• Communication Complexity of Distributed Convex Learning and Optimization
• Breaking Band: A Breakdown of High-performance Communication
• Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization
• Image Classification at Supercomputer Scale
• Parallel SGD: When does averaging help?
• AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training
• Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash
• Yet Another Accelerated SGD: ResNet-50 Training on ImageNet in 74.7 seconds
• Experiments on Parallel Training of Deep Neural Network using Model Averaging
• Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools
• 3LC: Lightweight and Effective Traffic Compression for Distributed Machine Learning
• Priority-based Parameter Propagation for Distributed DNN Training
• RedSync : Reducing Synchronization Traffic for Distributed Deep Learning
• Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to train
• Distributed TensorFlow with MPI
• Double Quantization for Communication-Efficient Distributed Optimization
• LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning
• MXNET-MPI: Embedding MPI parallelism in Parameter Server Task Model for scaling Deep Learning
• PowerAI DDL
• OD-SGD: One-step Delay Stochastic Gradient Descent for Distributed Training

Cited by in corpus (6)

• The Evolution of Distributed Systems for Graph Neural Networks and their Origin in Graph Processing and Deep Learning: A Survey
• DBS: Dynamic Batch Size For Distributed Deep Neural Network Training
• Convergence Analysis of Decentralized ASGD
• TensAIR: Real-Time Training of Neural Networks from Data-streams
• CD-SGD: Distributed Stochastic Gradient Descent with Compression and Delay Compensation
• SOAR: Minimizing Network Utilization with Bounded In-network Computing