Speeding up Convolutional Neural Networks By Exploiting the Sparsity of Rectifier Units
arXiv:1704.07724
Abstract
Rectifier neuron units (ReLUs) have been widely used in deep convolutional networks. An ReLU converts negative values to zeros, and does not change positive values, which leads to a high sparsity of neurons. In this work, we first examine the sparsity of the outputs of ReLUs in some popular deep convolutional architectures. And then we use the sparsity property of ReLUs to accelerate the calculation of convolution by skipping calculations of zero-valued neurons. The proposed sparse convolution algorithm achieves some speedup improvements on CPUs compared to the traditional matrix-matrix multiplication algorithm for convolution when the sparsity is not less than 0.9.
7 pages
References in corpus (5)
Cited by in corpus (15)
- Bayesian Compression for Deep Learning
- Hardware Acceleration of Sparse and Irregular Tensor Computations of ML Models: A Survey and Insights
- Sparse Deep Neural Network Exact Solutions
- SBNet: Sparse Blocks Network for Fast Inference
- GraphChallenge.org Sparse Deep Neural Network Performance
- Spatially Adaptive Inference with Stochastic Feature Sampling and Interpolation
- High Performance Convolution Using Sparsity and Patterns for Inference in Deep Convolutional Neural Networks
- Perceive, Attend, and Drive: Learning Spatial Attention for Safe Self-Driving
- PFA: Privacy-preserving Federated Adaptation for Effective Model Personalization
- Efficient Sparse-Dense Matrix-Matrix Multiplication on GPUs Using the Customized Sparse Storage Format
- Dual-side Sparse Tensor Core
- Accelerating convolutional neural network by exploiting sparsity on GPUs
- Cascaded Coarse-to-Fine Deep Kernel Networks for Efficient Satellite Image Change Detection
- Recurrent Residual Module for Fast Inference in Videos
- Adam Induces Implicit Weight Sparsity in Rectifier Neural Networks