Straggler-Resilient Distributed Machine Learning with Dynamic Backup Workers
arXiv:2102.06280
Abstract
With the increasing demand for large-scale training of machine learning models, consensus-based distributed optimization methods have recently been advocated as alternatives to the popular parameter server framework. In this paradigm, each worker maintains a local estimate of the optimal parameter vector, and iteratively updates it by waiting and averaging all estimates obtained from its neighbors, and then corrects it on the basis of its local dataset. However, the synchronization phase can be time consuming due to the need to wait for \textit{stragglers}, i.e., slower workers. An efficient way to mitigate this effect is to let each worker wait only for updates from the fastest neighbors before updating its local parameter. The remaining neighbors are called \textit{backup workers.} To minimize the globally training time over the network, we propose a fully distributed algorithm to dynamically determine the number of backup workers for each worker. We show that our algorithm achieves a linear speedup for convergence (i.e., convergence performance increases linearly with respect to the number of workers). We conduct extensive experiments on MNIST and CIFAR-10 to verify our theoretical results.
References in corpus (9)
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
- Revisiting Distributed Synchronous SGD
- Optimal algorithms for smooth and strongly convex distributed optimization in networks
- Communication Compression for Decentralized Training
- D: Decentralized Training over Decentralized Data
- Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
- Towards Flexible Device Participation in Federated Learning
- Toward Understanding the Impact of Staleness in Distributed Machine Learning