Improving compute efficacy frontiers with SliceOut
arXiv:2007.10909
Abstract
Pushing forward the compute efficacy frontier in deep learning is critical for tasks that require frequent model re-training or workloads that entail training a large number of models. We introduce SliceOut -- a dropout-inspired scheme designed to take advantage of GPU memory layout to train deep learning models faster without impacting final test accuracy. By dropping contiguous sets of units at random, our method realises training speedups through (1) fast memory access and matrix multiplication of smaller tensors, and (2) memory savings by avoiding allocating memory to zero units in weight gradients and activations. At test time, turning off SliceOut performs an implicit ensembling across a linear number of architectures that preserves test accuracy. We demonstrate 10-40% speedups and memory reduction with Wide ResNets, EfficientNets, and Transformer models, with minimal to no loss in accuracy. This leads to faster processing of large computational workloads overall, and significantly reduce the resulting energy consumption and CO2emissions.
References in corpus (13)
- Improving neural networks by preventing co-adaptation of feature detectors
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Deep Residual Learning for Image Recognition
- Improved Regularization of Convolutional Neural Networks with Cutout
- Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- DropEdge: Towards Deep Graph Convolutional Networks on Node Classification
- DropBlock: A regularization method for convolutional networks
- Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
- Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
- An Empirical Model of Large-Batch Training
- Learning Ordered Representations with Nested Dropout
- Efficient Object Localization Using Convolutional Networks