Channel Pruning for Accelerating Very Deep Neural Networks

Abstract

In this paper, we introduce a new channel pruning method to accelerate very deep convolutional neural networks.Given a trained CNN model, we propose an iterative two-step algorithm to effectively prune each layer, by a LASSO regression based channel selection and least square reconstruction. We further generalize this algorithm to multi-layer and multi-branch cases. Our method reduces the accumulated error and enhance the compatibility with various architectures. Our pruned VGG-16 achieves the state-of-the-art results by 5x speed-up along with only 0.3% increase of error. More importantly, our method is able to accelerate modern networks like ResNet, Xception and suffers only 1.4%, 1.0% accuracy loss under 2x speed-up respectively, which is significant. Code has been made publicly available.

To be appear at ICCV 2017

References in corpus (24)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Scikit-learn: Machine Learning in Python
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Going Deeper with Convolutions
• Compressing Deep Convolutional Networks using Vector Quantization
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• Pruning Filters for Efficient ConvNets
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Learning Structured Sparsity in Deep Neural Networks
• Aggregated Residual Transformations for Deep Neural Networks
• Xception: Deep Learning with Depthwise Separable Convolutions
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• Fast Training of Convolutional Networks through FFTs
• Fast Convolutional Nets With fbfft: A GPU Performance Evaluation
• Speed/accuracy trade-offs for modern convolutional object detectors
• Fast Algorithms for Convolutional Neural Networks
• Diversity Networks: Neural Network Compression Using Determinantal Point Processes
• Structured Pruning of Deep Convolutional Neural Networks
• Compact Deep Convolutional Neural Networks With Coarse Pruning
• Fast ConvNets Using Group-wise Brain Damage
• LCNN: Lookup-based Convolutional Neural Network

Cited by in corpus (101)

• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
• Pruning by Explaining: A Novel Criterion for Deep Neural Network Pruning
• ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• Slimmable Neural Networks
• Dynamic Channel Pruning: Feature Boosting and Suppression
• Bringing AI To Edge: From Deep Learning's Perspective
• Discrimination-aware Channel Pruning for Deep Neural Networks
• Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
• CondenseNet: An Efficient DenseNet using Learned Group Convolutions
• Balanced Sparsity for Efficient DNN Inference on GPU
• ProtoPShare: Prototype Sharing for Interpretable Image Classification and Similarity Discovery
• Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks
• A novel channel pruning method for deep neural network compression
• Accelerating Large Scale Real-Time GNN Inference using Channel Pruning
• Structured Probabilistic Pruning for Convolutional Neural Network Acceleration
• PruneTrain: Fast Neural Network Training by Dynamic Sparse Model Reconfiguration
• Spiking-YOLO: Spiking Neural Network for Energy-Efficient Object Detection
• DAIS: Automatic Channel Pruning via Differentiable Annealing Indicator Search
• Progressive Weight Pruning of Deep Neural Networks using ADMM
• Searching for Low-Bit Weights in Quantized Neural Networks
• Data-Driven Sparse Structure Selection for Deep Neural Networks
• PrivyNet: A Flexible Framework for Privacy-Preserving Deep Neural Network Training
• IGCV3: Interleaved Low-Rank Group Convolutions for Efficient Deep Neural Networks
• Learning to Prune Filters in Convolutional Neural Networks
• Channel Compression: Rethinking Information Redundancy among Channels in CNN Architecture
• Hybrid Pruning: Thinner Sparse Networks for Fast Inference on Edge Devices
• APQ: Joint Search for Network Architecture, Pruning and Quantization Policy
• Packing Sparse Convolutional Neural Networks for Efficient Systolic Array Implementations: Column Combining Under Joint Optimization
• Projection Convolutional Neural Networks for 1-bit CNNs via Discrete Back Propagation
• Few Shot Network Compression via Cross Distillation
• Hardware-Centric AutoML for Mixed-Precision Quantization
• Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentation
• HALOC: Hardware-Aware Automatic Low-Rank Compression for Compact Neural Networks
• Accelerating Sparse Deep Neural Networks
• SBNet: Sparse Blocks Network for Fast Inference
• An Experimental Study of the Impact of Pre-training on the Pruning of a Convolutional Neural Network
• Fine-Grained Neural Architecture Search
• Shift-based Primitives for Efficient Convolutional Neural Networks
• Pruning via Iterative Ranking of Sensitivity Statistics
• Heavy Tails in SGD and Compressibility of Overparametrized Neural Networks
• Neuron Merging: Compensating for Pruned Neurons
• GeneCAI: Genetic Evolution for Acquiring Compact AI
• BLK-REW: A Unified Block-based DNN Pruning Framework using Reweighted Regularization Method
• Taxonomy of Saliency Metrics for Channel Pruning
• Compact Global Descriptor for Neural Networks
• Squeezed Deep 6DoF Object Detection Using Knowledge Distillation
• Structured Pruning for Efficient ConvNets via Incremental Regularization
• Learning Versatile Convolution Filters for Efficient Visual Recognition
• Accelerating Deep Neural Networks with Spatial Bottleneck Modules
• Hu-Fu: Hardware and Software Collaborative Attack Framework against Neural Networks
• Fractional Skipping: Towards Finer-Grained Dynamic CNN Inference
• CNNPruner: Pruning Convolutional Neural Networks with Visual Analytics
• Enabling Retrain-free Deep Neural Network Pruning using Surrogate Lagrangian Relaxation
• Exploring Gradient Flow Based Saliency for DNN Model Compression
• Pruning Deep Neural Networks using Partial Least Squares
• Characterising Across-Stack Optimisations for Deep Convolutional Neural Networks
• Demystifying Neural Network Filter Pruning
• Stochastic Downsampling for Cost-Adjustable Inference and Improved Regularization in Convolutional Networks
• Unifying and Merging Well-trained Deep Neural Networks for Inference Stage
• How is Contrast Encoded in Deep Neural Networks?
• CNN Acceleration by Low-rank Approximation with Quantized Factors
• Cramnet: Layer-wise Deep Neural Network Compression with Knowledge Transfer from a Teacher Network
• Doubly Nested Network for Resource-Efficient Inference
• Real-Time Object Tracking via Meta-Learning: Efficient Model Adaptation and One-Shot Channel Pruning
• PydMobileNet: Improved Version of MobileNets with Pyramid Depthwise Separable Convolution
• CAE-ADMM: Implicit Bitrate Optimization via ADMM-based Pruning in Compressive Autoencoders
• Generalized Bayesian Posterior Expectation Distillation for Deep Neural Networks
• Efficient Neural Network Compression
• Merging and Evolution: Improving Convolutional Neural Networks for Mobile Applications
• Recent Advances in Efficient Computation of Deep Convolutional Neural Networks
• Channel-Level Variable Quantization Network for Deep Image Compression
• Graph-Adaptive Pruning for Efficient Inference of Convolutional Neural Networks
• CompNet: Neural networks growing via the compact network morphism
• Recent Advances in Convolutional Neural Network Acceleration
• Improving Efficiency in Neural Network Accelerator Using Operands Hamming Distance optimization
• Diagonalwise Refactorization: An Efficient Training Method for Depthwise Convolutions
• Plug-in, Trainable Gate for Streamlining Arbitrary Neural Networks
• Studying the Plasticity in Deep Convolutional Neural Networks using Random Pruning
• Neural Rejuvenation: Improving Deep Network Training by Enhancing Computational Resource Utilization
• Quantization Mimic: Towards Very Tiny CNN for Object Detection
• A Main/Subsidiary Network Framework for Simplifying Binary Neural Network
• A Layer Decomposition-Recomposition Framework for Neuron Pruning towards Accurate Lightweight Networks
• CondenseNet V2: Sparse Feature Reactivation for Deep Networks
• AutoPruning for Deep Neural Network with Dynamic Channel Masking
• MultiTASC++: A Continuously Adaptive Scheduler for Edge-Based Multi-Device Cascade Inference
• BPPSA: Scaling Back-propagation by Parallel Scan Algorithm
• Recurrent Residual Module for Fast Inference in Videos
• Softer Pruning, Incremental Regularization
• Feature Statistics Guided Efficient Filter Pruning
• Auto-Split: A General Framework of Collaborative Edge-Cloud AI
• Sub-bit Neural Networks: Learning to Compress and Accelerate Binary Neural Networks
• Rethinking Lightweight Convolutional Neural Networks for Efficient and High-quality Pavement Crack Detection
• Auto Deep Compression by Reinforcement Learning Based Actor-Critic Structure
• Efficient Semantic Segmentation for Visual Bird's-eye View Interpretation
• DAC: Data-free Automatic Acceleration of Convolutional Networks
• Writer-Aware CNN for Parsimonious HMM-Based Offline Handwritten Chinese Text Recognition
• SdcNet: A Computation-Efficient CNN for Object Recognition
• HALO: Learning to Prune Neural Networks with Shrinkage
• One-Cycle Pruning: Pruning ConvNets Under a Tight Training Budget