An Entropy-based Pruning Method for CNN Compression

Abstract

This paper aims to simultaneously accelerate and compress off-the-shelf CNN models via filter pruning strategy. The importance of each filter is evaluated by the proposed entropy-based method first. Then several unimportant filters are discarded to get a smaller CNN model. Finally, fine-tuning is adopted to recover its generalization ability which is damaged during filter pruning. Our method can reduce the size of intermediate activations, which would dominate most memory footprint during model training stage but is less concerned in previous compression methods. Experiments on the ILSVRC-12 benchmark demonstrate the effectiveness of our method. Compared with previous filter importance evaluation criteria, our entropy-based method obtains better performance. We achieve 3.3x speed-up and 16.64x compression on VGG-16, 1.54x acceleration and 1.47x compression on ResNet-50, both with about 1% top-5 accuracy decrease.

References in corpus (14)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Improving neural networks by preventing co-adaptation of feature detectors
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Going Deeper with Convolutions
• Compressing Deep Convolutional Networks using Vector Quantization
• Fully Convolutional Networks for Semantic Segmentation
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• Predicting Parameters in Deep Learning
• Pruning Filters for Efficient ConvNets
• Rich feature hierarchies for accurate object detection and semantic segmentation
• Quantized Convolutional Neural Networks for Mobile Devices

Cited by in corpus (20)

• Structured Pruning for Deep Convolutional Neural Networks: A survey
• Pruning Deep Convolutional Neural Networks Architectures with Evolution Strategy
• 2PFPCE: Two-Phase Filter Pruning Based on Conditional Entropy
• Pruning Algorithms to Accelerate Convolutional Neural Networks for Edge Applications: A Survey
• Complexity-Driven CNN Compression for Resource-constrained Edge AI
• AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Rates
• Can Unstructured Pruning Reduce the Depth in Deep Neural Networks?
• Analysing Deep Reinforcement Learning Agents Trained with Domain Randomisation
• Sauron U-Net: Simple automated redundancy elimination in medical image segmentation via filter pruning
• Convolution-Weight-Distribution Assumption: Rethinking the Criteria of Channel Pruning
• DARC: Differentiable ARchitecture Compression
• Overcoming Catastrophic Forgetting by Neuron-level Plasticity Control
• UCP: Uniform Channel Pruning for Deep Convolutional Neural Networks Compression and Acceleration
• Deep Asymmetric Networks with a Set of Node-wise Variant Activation Functions
• Efficient architecture for deep neural networks with heterogeneous sensitivity
• Studying the Plasticity in Deep Convolutional Neural Networks using Random Pruning
• Directed-Weighting Group Lasso for Eltwise Blocked CNN Pruning
• Channel Pruning Guided by Spatial and Channel Attention for DNNs in Intelligent Edge Computing
• Towards Real-Time DNN Inference on Mobile Platforms with Model Pruning and Compiler Optimization
• Blending Pruning Criteria for Convolutional Neural Networks