Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGD
arXiv:1810.08313
Abstract
Large-scale machine learning training, in particular distributed stochastic gradient descent, needs to be robust to inherent system variability such as node straggling and random communication delays. This work considers a distributed training framework where each worker node is allowed to perform local model updates and the resulting models are averaged periodically. We analyze the true speed of error convergence with respect to wall-clock time (instead of the number of iterations), and analyze how it is affected by the frequency of averaging. The main contribution is the design of AdaComm, an adaptive communication strategy that starts with infrequent averaging to save communication delay and improve convergence speed, and then increases the communication frequency in order to achieve a low error floor. Rigorous experiments on training deep neural networks show that AdaComm can take less time than fully synchronous SGD, and still reach the same final training loss.
arXiv commentAccepted to SysML 2019
References in corpus (15)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Don't Decay the Learning Rate, Increase the Batch Size
- Optimal Distributed Online Prediction using Mini-Batches
- Don't Use Large Mini-Batches, Use Local SGD
- Local SGD Converges Fast and Communicates Little
- Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
- PipeDream: Fast and Efficient Pipeline Parallel DNN Training
- Parallel SGD: When does averaging help?
- SparkNet: Training Deep Networks in Spark
- Experiments on Parallel Training of Deep Neural Network using Model Averaging
- Slow and Stale Gradients Can Win the Race: Error-Runtime Trade-offs in Distributed SGD
- Coupling Adaptive Batch Sizes with Learning Rates
- Parle: parallelizing stochastic gradient descent
Cited by in corpus (13)
- Improving Federated Learning Personalization via Model Agnostic Meta Learning
- Communication-Efficient Edge AI: Algorithms and Systems
- Accelerating Generalized Linear Models with MLWeaving: A One-Size-Fits-All System for Any-precision Learning (Technical Report)
- Differentially Private Federated Learning for Resource-Constrained Internet of Things
- Communication-Efficient Distributed Stochastic AUC Maximization with Deep Neural Networks
- Taming Momentum in a Distributed Asynchronous Environment
- Gradient Descent with Compressed Iterates
- RingFed: Reducing Communication Costs in Federated Learning on Non-IID Data
- Communication-Efficient Distributed SVD via Local Power Iterations
- Distributed Machine Learning through Heterogeneous Edge Systems
- Citadel: Protecting Data Privacy and Model Confidentiality for Collaborative Learning with SGX
- Daydream: Accurately Estimating the Efficacy of Optimizations for DNN Training
- Trade-offs of Local SGD at Scale: An Empirical Study