Taming Unbalanced Training Workloads in Deep Learning with Partial Collective Operations

Abstract

Load imbalance pervasively exists in distributed deep learning training systems, either caused by the inherent imbalance in learned tasks or by the system itself. Traditional synchronous Stochastic Gradient Descent (SGD) achieves good accuracy for a wide variety of tasks, but relies on global synchronization to accumulate the gradients at every training step. In this paper, we propose eager-SGD, which relaxes the global synchronization for decentralized accumulation. To implement eager-SGD, we propose to use two partial collectives: solo and majority. With solo allreduce, the faster processes contribute their gradients eagerly without waiting for the slower processes, whereas with majority allreduce, at least half of the participants must contribute gradients before continuing, all without using a central parameter server. We theoretically prove the convergence of the algorithms and describe the partial collectives in detail. Experimental results on load-imbalanced environments (CIFAR-10, ImageNet, and UCF101 datasets) show that eager-SGD achieves 1.27x speedup over the state-of-the-art synchronous SGD, without losing accuracy.

Published in Proceedings of the 25th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP'20), pp. 45-61. 2020, Best Paper Nomination

References in corpus (12)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Improved Regularization of Convolutional Neural Networks with Cutout
• Horovod: fast and easy distributed deep learning in TensorFlow
• Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• Staleness-aware Async-SGD for Distributed Deep Learning
• Delving Deeper into Convolutional Networks for Learning Video Representations
• GossipGraD: Scalable Deep Learning using Gossip Communication based Asynchronous Gradient Descent
• How to scale distributed deep learning?
• SparCML: High-Performance Sparse Communication for Machine Learning
• Model Accuracy and Runtime Tradeoff in Distributed Deep Learning:A Systematic Study

Cited by in corpus (11)

• Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines
• Near-Optimal Sparse Allreduce for Distributed Deep Learning
• Flare: Flexible In-Network Allreduce
• Hanayo: Harnessing Wave-like Pipeline Parallelism for Enhanced Large Model Training Efficiency
• AutoDDL: Automatic Distributed Deep Learning with Near-Optimal Bandwidth Cost
• An In-Depth Analysis of the Slingshot Interconnect
• HPCClusterScape: Increasing Transparency and Efficiency of Shared High-Performance Computing Clusters for Large-scale AI Models
• Breaking (Global) Barriers in Parallel Stochastic Optimization with Wait-Avoiding Group Averaging
• Consistent Lock-free Parallel Stochastic Gradient Descent for Fast and Stable Convergence
• CD-SGD: Distributed Stochastic Gradient Descent with Compression and Delay Compensation
• Asynchronous Fully-Decentralized SGD in the Cluster-Based Model