Accelerating Distributed ML Training via Selective Synchronization

Abstract

In distributed training, deep neural networks (DNNs) are launched over multiple workers concurrently and aggregate their local updates on each step in bulk-synchronous parallel (BSP) training. However, BSP does not linearly scale-out due to high communication cost of aggregation. To mitigate this overhead, alternatives like Federated Averaging (FedAvg) and Stale-Synchronous Parallel (SSP) either reduce synchronization frequency or eliminate it altogether, usually at the cost of lower final accuracy. In this paper, we present \texttt{SelSync}, a practical, low-overhead method for DNN training that dynamically chooses to incur or avoid communication at each step either by calling the aggregation op or applying local updates based on their significance. We propose various optimizations as part of \texttt{SelSync} to improve convergence in the context of \textit{semi-synchronous} training. Our system converges to the same or better accuracy than BSP while reducing training time by up to 14$\times$.

References in corpus (22)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Federated Learning: Challenges, Methods, and Future Directions
• An overview of gradient descent optimization algorithms
• Federated Learning with Non-IID Data
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• Mixed Precision Training
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization
• An Empirical Model of Large-Batch Training
• Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
• Critical Learning Periods in Deep Neural Networks
• Optimizing Network Performance for Distributed DNN Training on GPU Clusters: ImageNet/AlexNet Training in 1.5 Minutes
• Large batch size training of neural networks with adversarial training and second-order information
• Taming Resource Heterogeneity In Distributed ML Training With Dynamic Batching
• AdaScale SGD: A User-Friendly Algorithm for Distributed Training
• On the Utility of Gradient Compression in Distributed Training Systems
• Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification
• ScaDLES: Scalable Deep Learning over Streaming data at the Edge
• GraVAC: Adaptive Compression for Communication-Efficient Distributed DL Training

Cited by in corpus (2)

• Flexible Communication for Optimal Distributed Learning over Unpredictable Networks
• When Less is More: Achieving Faster Convergence in Distributed Edge Machine Learning