MXNET-MPI: Embedding MPI parallelism in Parameter Server Task Model for scaling Deep Learning
arXiv:1801.03855
Abstract
Existing Deep Learning frameworks exclusively use either Parameter Server(PS) approach or MPI parallelism. In this paper, we discuss the drawbacks of such approaches and propose a generic framework supporting both PS and MPI programming paradigms, co-existing at the same time. The key advantage of the new model is to embed the scaling benefits of MPI parallelism into the loosely coupled PS task model. Apart from providing a practical usage model of MPI in cloud, such framework allows for novel communication avoiding algorithms that do parameter averaging in Stochastic Gradient Descent(SGD) approaches. We show how MPI and PS models can synergestically apply algorithms such as Elastic SGD to improve the rate of convergence against existing approaches. These new algorithms directly help scaling SGD clusterwide. Further, we also optimize the critical component of the framework, namely global aggregation or allreduce using a novel concept of tensor collectives. These treat a group of vectors on a node as a single object allowing for the existing single vector algorithms to be directly applicable. We back our claims with sufficient emperical evidence using large scale ImageNet 1K data. Our framework is built upon MXNET but the design is generic and can be adapted to other popular DL infrastructures.
References in corpus (6)
- TensorFlow: A system for large-scale machine learning
- An overview of gradient descent optimization algorithms
- MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
- HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
- Revisiting Distributed Synchronous SGD
- Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization
Cited by in corpus (5)
- A Survey on Distributed Machine Learning
- Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform
- Communication optimization strategies for distributed deep neural network training: A survey
- HeterPS: Distributed Deep Learning With Reinforcement Learning Based Scheduling in Heterogeneous Environments
- Parallax: Sparsity-aware Data Parallel Training of Deep Neural Networks