2PFPCE: Two-Phase Filter Pruning Based on Conditional Entropy

Abstract

Deep Convolutional Neural Networks~(CNNs) offer remarkable performance of classifications and regressions in many high-dimensional problems and have been widely utilized in real-word cognitive applications. However, high computational cost of CNNs greatly hinder their deployment in resource-constrained applications, real-time systems and edge computing platforms. To overcome this challenge, we propose a novel filter-pruning framework, two-phase filter pruning based on conditional entropy, namely \textit{2PFPCE}, to compress the CNN models and reduce the inference time with marginal performance degradation. In our proposed method, we formulate filter pruning process as an optimization problem and propose a novel filter selection criteria measured by conditional entropy. Based on the assumption that the representation of neurons shall be evenly distributed, we also develop a maximum-entropy filter freeze technique that can reduce over fitting. Two filter pruning strategies -- global and layer-wise strategies, are compared. Our experiment result shows that combining these two strategies can achieve a higher neural network compression ratio than applying only one of them under the same accuracy drop threshold. Two-phase pruning, that is, combining both global and layer-wise strategies, achieves 10 X FLOPs reduction and 46% inference time reduction on VGG-16, with 2% accuracy drop.

arXiv comment8 pages, 6 figures

References in corpus (14)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Estimating Mutual Information
• Do Deep Nets Really Need to be Deep?
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• Compressing Deep Convolutional Networks using Vector Quantization
• Opening the Black Box of Deep Neural Networks via Information
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• Predicting Parameters in Deep Learning
• Pruning Filters for Efficient ConvNets
• Learning Structured Sparsity in Deep Neural Networks
• Convolutional neural networks with low-rank regularization
• Fast Convolutional Nets With fbfft: A GPU Performance Evaluation
• An Entropy-based Pruning Method for CNN Compression

Cited by in corpus (19)

• HRel: Filter Pruning based on High Relevance between Activation Maps and Class Labels
• AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Rates
• Can Unstructured Pruning Reduce the Depth in Deep Neural Networks?
• Non-Structured DNN Weight Pruning -- Is It Beneficial in Any Platform?
• YOLObile: Real-Time Object Detection on Mobile Devices via Compression-Compilation Co-Design
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• SS-Auto: A Single-Shot, Automatic Structured Weight Pruning Framework of DNNs with Ultra-High Efficiency
• RTMobile: Beyond Real-Time Mobile Acceleration of RNNs for Speech Recognition
• BLK-REW: A Unified Block-based DNN Pruning Framework using Reweighted Regularization Method
• A SOT-MRAM-based Processing-In-Memory Engine for Highly Compressed DNN Implementation
• ResNet Can Be Pruned 60x: Introducing Network Purification and Unused Path Removal (P-RM) after Weight Pruning
• An Image Enhancing Pattern-based Sparsity for Real-time Inference on Mobile Devices
• Achieving Real-Time LiDAR 3D Object Detection on a Mobile Device
• NPAS: A Compiler-aware Framework of Unified Network Pruning and Architecture Search for Beyond Real-Time Mobile Acceleration
• Manipulating Identical Filter Redundancy for Efficient Pruning on Deep and Complicated CNN
• Achieving on-Mobile Real-Time Super-Resolution with Neural Architecture and Pruning Search
• A Privacy-Preserving-Oriented DNN Pruning and Mobile Acceleration Framework
• Channel Pruning Guided by Spatial and Channel Attention for DNNs in Intelligent Edge Computing
• GRIM: A General, Real-Time Deep Learning Inference Framework for Mobile Devices based on Fine-Grained Structured Weight Sparsity