Thanks for Nothing: Predicting Zero-Valued Activations with Lightweight Convolutional Neural Networks
arXiv:1909.07636
Abstract
Convolutional neural networks (CNNs) introduce state-of-the-art results for various tasks with the price of high computational demands. Inspired by the observation that spatial correlation exists in CNN output feature maps (ofms), we propose a method to dynamically predict whether ofm activations are zero-valued or not according to their neighboring activation values, thereby avoiding zero-valued activations and reducing the number of convolution operations. We implement the zero activation predictor (ZAP) with a lightweight CNN, which imposes negligible overheads and is easy to deploy on existing models. ZAPs are trained by mimicking hidden layer ouputs; thereby, enabling a parallel and label-free training. Furthermore, without retraining, each ZAP can be tuned to a different operating point trading accuracy for MAC reduction.
References in corpus (9)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Neural Architecture Search with Reinforcement Learning
- An Analysis of Deep Neural Network Models for Practical Applications
- Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
- PerforatedCNNs: Acceleration through Elimination of Redundant Convolutions
- Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform
- Spatial Correlation and Value Prediction in Convolutional Neural Networks
- Divide and Conquer: Leveraging Intermediate Feature Representations for Quantized Training of Neural Networks