Gear Training: A new way to implement high-performance model-parallel training
arXiv:1806.03925
Abstract
The training of Deep Neural Networks usually needs tremendous computing resources. Therefore many deep models are trained in large cluster instead of single machine or GPU. Though major researchs at present try to run whole model on all machines by using asynchronous asynchronous stochastic gradient descent (ASGD), we present a new approach to train deep model parallely -- split the model and then seperately train different parts of it in different speed.
References in corpus (7)
- Deep Residual Learning for Image Recognition
- Revisiting Distributed Synchronous SGD
- Wide & Deep Learning for Recommender Systems
- BigDL: A Distributed Deep Learning Framework for Big Data
- Distributed TensorFlow with MPI
- AMPNet: Asynchronous Model-Parallel Training for Dynamic Neural Networks
- Stochastic Gradient Descent on Highly-Parallel Architectures