Elastic Consistency: A General Consistency Model for Distributed Stochastic Gradient Descent
arXiv:2001.05918
Abstract
Machine learning has made tremendous progress in recent years, with models matching or even surpassing humans on a series of specialized tasks. One key element behind the progress of machine learning in recent years has been the ability to train machine learning models in large-scale distributed shared-memory and message-passing environments. Many of these models are trained employing variants of stochastic gradient descent (SGD) based optimization. In this paper, we introduce a general consistency condition covering communication-reduced and asynchronous distributed SGD implementations. Our framework, called elastic consistency enables us to derive convergence bounds for a variety of distributed SGD methods used in practice to train large-scale machine learning models. The proposed framework de-clutters the implementation-specific convergence analysis and provides an abstraction to derive convergence bounds. We utilize the framework to analyze a sparsification scheme for distributed SGD methods in an asynchronous setting for convex and non-convex objectives. We implement the distributed SGD variant to train deep CNN models in an asynchronous shared-memory setting. Empirical results show that error-feedback may not necessarily help in improving the convergence of sparsified asynchronous distributed SGD, which corroborates an insight suggested by our convergence analysis.
References in corpus (19)
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Wide Residual Networks
- HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
- Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
- QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
- Horovod: fast and easy distributed deep learning in TensorFlow
- Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization
- Sparsified SGD with Memory
- Don't Use Large Mini-Batches, Use Local SGD
- Local SGD Converges Fast and Communicates Little
- Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
- Error Feedback Fixes SignSGD and other Gradient Compression Schemes
- Taming the Wild: A Unified Analysis of Hogwild!-Style Algorithms
- SGD and Hogwild! Convergence Without the Bounded Gradients Assumption
- The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Communication
- Graph Oracle Models, Lower Bounds, and Gaps for Parallel Stochastic Optimization
- Priority-based Parameter Propagation for Distributed DNN Training
- Fault Tolerance in Iterative-Convergent Machine Learning
- MixML: A Unified Analysis of Weakly Consistent Parallel Learning