Reducing the Training Time of Neural Networks by Partitioning
arXiv:1511.02954
Abstract
This paper presents a new method for pre-training neural networks that can decrease the total training time for a neural network while maintaining the final performance, which motivates its use on deep neural networks. By partitioning the training task in multiple training subtasks with sub-models, which can be performed independently and in parallel, it is shown that the size of the sub-models reduces almost quadratically with the number of subtasks created, quickly scaling down the sub-models used for the pre-training. The sub-models are then merged to provide a pre-trained initial set of weights for the original model. The proposed method is independent of the other aspects of the training, such as architecture of the neural network, training method, and objective, making it compatible with a wide range of existing approaches. The speedup without loss of performance is validated experimentally on MNIST and on CIFAR10 data sets, also showing that even performing the subtasks sequentially can decrease the training time. Moreover, we show that larger models may present higher speedups and conjecture about the benefits of the method in distributed learning systems.
Figure 2b has lower quality due to file size constraints
References in corpus (11)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Distilling the Knowledge in a Neural Network
- Going Deeper with Convolutions
- Deep Learning with Limited Numerical Precision
- cuDNN: Efficient Primitives for Deep Learning
- One weird trick for parallelizing convolutional neural networks
- The Loss Surfaces of Multilayer Networks
- Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
- Fast Convolutional Nets With fbfft: A GPU Performance Evaluation
- Net2Net: Accelerating Learning via Knowledge Transfer
- On the saddle point problem for non-convex optimization