Accelerating CNN Training by Pruning Activation Gradients
arXiv:1908.00173
Abstract
Sparsification is an efficient approach to accelerate CNN inference, but it is challenging to take advantage of sparsity in training procedure because the involved gradients are dynamically changed. Actually, an important observation shows that most of the activation gradients in back-propagation are very close to zero and only have a tiny impact on weight-updating. Hence, we consider pruning these very small gradients randomly to accelerate CNN training according to the statistical distribution of activation gradients. Meanwhile, we theoretically analyze the impact of pruning algorithm on the convergence. The proposed approach is evaluated on AlexNet and ResNet-\{18, 34, 50, 101, 152\} with CIFAR-\{10, 100\} and ImageNet datasets. Experimental results show that our training approach could substantially achieve up to speedups at back-propagation stage with negligible accuracy loss.
arXiv commentaccepted by ECCV 2020
References in corpus (14)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Caffe: Convolutional Architecture for Fast Feature Embedding
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
- Mixed Precision Training
- TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
- Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
- Exploring the Regularity of Sparse Structure in Convolutional Neural Networks
- Learning Intrinsic Sparse Structures within Long Short-Term Memory
- meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting
- Minimal Effort Back Propagation for Convolutional Neural Networks
- Memorized Sparse Backpropagation
- RePr: Improved Training of Convolutional Filters