Theano-MPI: a Theano-based Distributed Training Framework
arXiv:1605.08325
Abstract
We develop a scalable and extendable training framework that can utilize GPUs across nodes in a cluster and accelerate the training of deep learning models based on data parallelism. Both synchronous and asynchronous training are implemented in our framework, where parameter exchange among GPUs is based on CUDA-aware MPI. In this report, we analyze the convergence and capability of the framework to reduce training time when scaling the synchronous training of AlexNet and GoogLeNet from 2 GPUs to 8 GPUs. In addition, we explore novel ways to reduce the communication overhead caused by exchanging parameters. Finally, we release the framework as open-source for further research on distributed deep learning
References in corpus (8)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Deep Residual Learning for Image Recognition
- Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
- MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
- Going Deeper with Convolutions
- One weird trick for parallelizing convolutional neural networks
- Theano-based Large-Scale Visual Recognition with Multiple GPUs
- Purine: A bi-graph based deep learning framework