Optimizing Network Performance for Distributed DNN Training on GPU Clusters: ImageNet/AlexNet Training in 1.5 Minutes

Abstract

It is important to scale out deep neural network (DNN) training for reducing model training time. The high communication overhead is one of the major performance bottlenecks for distributed DNN training across multiple GPUs. Our investigations have shown that popular open-source DNN systems could only achieve 2.5 speedup ratio on 64 GPUs connected by 56 Gbps network. To address this problem, we propose a communication backend named GradientFlow for distributed DNN training, and employ a set of network optimization techniques. First, we integrate ring-based allreduce, mixed-precision training, and computation/communication overlap into GradientFlow. Second, we propose lazy allreduce to improve network throughput by fusing multiple communication operations into a single one, and design coarse-grained sparse communication to reduce network traffic by only transmitting important gradient chunks. When training ImageNet/AlexNet on 512 GPUs, our approach achieves 410.2 speedup ratio and completes 95-epoch training in 1.5 minutes, which outperforms existing approaches.

References in corpus (14)

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• cuDNN: Efficient Primitives for Deep Learning
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• Mixed Precision Training
• Don't Decay the Learning Rate, Increase the Batch Size
• Horovod: fast and easy distributed deep learning in TensorFlow
• Deep Image: Scaling up Image Recognition
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes
• Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash
• Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to train
• TicTac: Accelerating Distributed Deep Learning with Communication Scheduling

Cited by in corpus (15)

• Knowledge Distillation: A Survey
• MLPerf Training Benchmark
• Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications
• Understanding Top-k Sparsification in Distributed Deep Learning
• SenseCare: A Research Platform for Medical Image Informatics and Interactive 3D Visualization
• Communication-Efficient Edge AI: Algorithms and Systems
• Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts
• Heterogeneity-Aware Asynchronous Decentralized Training
• ILASR: Privacy-Preserving Incremental Learning for Automatic Speech Recognition at Production Scale
• EmbRace: Accelerating Sparse Communication for Distributed Training of NLP Neural Networks
• Accelerating Distributed ML Training via Selective Synchronization
• VirtualFlow: Decoupling Deep Learning Models from the Underlying Hardware
• Crossover-SGD: A gossip-based communication in distributed deep learning for alleviating large mini-batch problem and enhancing scalability
• Flexible Communication for Optimal Distributed Learning over Unpredictable Networks
• OD-SGD: One-step Delay Stochastic Gradient Descent for Distributed Training