Pruning Filters for Efficient ConvNets

Abstract

The success of CNNs in various applications is accompanied by a significant increase in the computation and parameter storage costs. Recent efforts toward reducing these overheads involve pruning and compressing the weights of various layers without hurting original accuracy. However, magnitude-based pruning of weights reduces a significant number of parameters from the fully connected layers and may not adequately reduce the computation costs in the convolutional layers due to irregular sparsity in the pruned networks. We present an acceleration method for CNNs, where we prune filters from CNNs that are identified as having a small effect on the output accuracy. By removing whole filters in the network together with their connecting feature maps, the computation costs are reduced significantly. In contrast to pruning weights, this approach does not result in sparse connectivity patterns. Hence, it does not need the support of sparse convolution libraries and can work with existing efficient BLAS libraries for dense matrix multiplications. We show that even simple filter pruning techniques can reduce inference costs for VGG-16 by up to 34% and ResNet-110 by up to 38% on CIFAR10 while regaining close to the original accuracy by retraining the networks.

Published as a conference paper at ICLR 2017

References in corpus (6)

• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Fast Training of Convolutional Networks through FFTs
• XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
• Deep Networks with Stochastic Depth
• Structured Pruning of Deep Convolutional Neural Networks

Cited by in corpus (272)

• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Object Detection in 20 Years: A Survey
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Channel Pruning for Accelerating Very Deep Neural Networks
• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
• Reducing Transformer Depth on Demand with Structured Dropout
• Learning Efficient Convolutional Networks through Network Slimming
• Deep Learning for Generic Object Detection: A Survey
• Slimmable Neural Networks
• Exploring the Regularity of Sparse Structure in Convolutional Neural Networks
• Ablation Studies in Artificial Neural Networks
• Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inference
• CornerNet-Lite: Efficient Keypoint Based Object Detection
• An Entropy-based Pruning Method for CNN Compression
• AutoSlim: Towards One-Shot Architecture Search for Channel Numbers
• Stabilizing the Lottery Ticket Hypothesis
• ICNet for Real-Time Semantic Segmentation on High-Resolution Images
• Training with Quantization Noise for Extreme Model Compression
• One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers
• Terminal Brain Damage: Exposing the Graceless Degradation in Deep Neural Networks Under Hardware Fault Attacks
• Review of data analysis in vision inspection of power lines with an in-depth discussion of deep learning technology
• CondenseNet: An Efficient DenseNet using Learned Group Convolutions
• Revisiting the Importance of Individual Units in CNNs via Ablation
• Feature-map-level Online Adversarial Knowledge Distillation
• Deep Neural Network Approximation for Custom Hardware: Where We've Been, Where We're Going
• A Low Effort Approach to Structured CNN Design Using PCA
• Proving the Lottery Ticket Hypothesis: Pruning is All You Need
• How Can We Be So Dense? The Benefits of Using Highly Sparse Representations
• Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon
• Compacting Deep Neural Networks for Internet of Things: Methods and Applications
• Crossbar-aware neural network pruning
• Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
• ModelDiff: Testing-Based DNN Similarity Comparison for Model Reuse Detection
• MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without Tricks
• Attention-based Convolutional Autoencoders for 3D-Variational Data Assimilation
• 2PFPCE: Two-Phase Filter Pruning Based on Conditional Entropy
• RepVGG: Making VGG-style ConvNets Great Again
• Reweighted Proximal Pruning for Large-Scale Language Representation
• Are Sixteen Heads Really Better than One?
• Only Train Once: A One-Shot Neural Network Training And Pruning Framework
• Pruning Algorithms to Accelerate Convolutional Neural Networks for Edge Applications: A Survey
• YOLO-LITE: A Real-Time Object Detection Algorithm Optimized for Non-GPU Computers
• Compact Deep Convolutional Neural Networks With Coarse Pruning
• More is Less: A More Complicated Network with Less Inference Complexity
• Knowledge Distillation via Route Constrained Optimization
• HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision
• Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks
• A Learning Framework for n-bit Quantized Neural Networks toward FPGAs
• SCSP: Spectral Clustering Filter Pruning with Soft Self-adaption Manners
• ChipNet: Budget-Aware Pruning with Heaviside Continuous Approximations
• Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracy
• Non-Structured DNN Weight Pruning -- Is It Beneficial in Any Platform?
• PrivyNet: A Flexible Framework for Privacy-Preserving Deep Neural Network Training
• Optimal Lottery Tickets via SubsetSum: Logarithmic Over-Parameterization is Sufficient
• Task dependent Deep LDA pruning of neural networks
• CHIP: CHannel Independence-based Pruning for Compact Neural Networks
• Provable Filter Pruning for Efficient Neural Networks
• Model Slicing for Supporting Complex Analytics with Elastic Inference Cost and Resource Constraints
• A Programmable Approach to Neural Network Compression
• Lightweight Residual Densely Connected Convolutional Neural Network
• Structural Compression of Convolutional Neural Networks
• Handcrafted Backdoors in Deep Neural Networks
• Distilling Critical Paths in Convolutional Neural Networks
• ZeroQ: A Novel Zero Shot Quantization Framework
• IPGuard: Protecting Intellectual Property of Deep Neural Networks via Fingerprinting the Classification Boundary
• Anchor Pruning for Object Detection
• HAWQV3: Dyadic Neural Network Quantization
• Few Shot Network Compression via Cross Distillation
• I-BERT: Integer-only BERT Quantization
• Post-Training Piecewise Linear Quantization for Deep Neural Networks
• Modeling of Pruning Techniques for Deep Neural Networks Simplification
• The Search for Sparse, Robust Neural Networks
• One Person, One Model, One World: Learning Continual User Representation without Forgetting
• Taxonomy and Evaluation of Structured Compression of Convolutional Neural Networks
• Knowledge Distillation for End-to-End Person Search
• Interpreting Convolutional Neural Networks Through Compression
• Path homologies of deep feedforward networks
• SS-Auto: A Single-Shot, Automatic Structured Weight Pruning Framework of DNNs with Ultra-High Efficiency
• An Experimental Study of the Impact of Pre-training on the Pruning of a Convolutional Neural Network
• Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Model
• Real-time CNN-based Segmentation Architecture for Ball Detection in a Single View Setup
• Bringing Giant Neural Networks Down to Earth with Unlabeled Data
• Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure
• Achieving Adversarial Robustness via Sparsity
• Layer Pruning via Fusible Residual Convolutional Block for Deep Neural Networks
• Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot
• EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruning
• DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures
• Trained Rank Pruning for Efficient Deep Neural Networks
• Knowledge Squeezed Adversarial Network Compression
• Pruning via Iterative Ranking of Sensitivity Statistics
• PruneNet: Channel Pruning via Global Importance
• CGaP: Continuous Growth and Pruning for Efficient Deep Learning
• Equivalent and Approximate Transformations of Deep Neural Networks
• GeneCAI: Genetic Evolution for Acquiring Compact AI
• Enabling Large Neural Networks on Tiny Microcontrollers with Swapping
• Sparse Weight Activation Training
• Deep Hashing with Category Mask for Fast Video Retrieval
• Patch Slimming for Efficient Vision Transformers
• Rethinking Class-Discrimination Based CNN Channel Pruning
• Taxonomy of Saliency Metrics for Channel Pruning
• Convolution-Weight-Distribution Assumption: Rethinking the Criteria of Channel Pruning
• Joint Pruning & Quantization for Extremely Sparse Neural Networks
• Ensemble Knowledge Distillation for Learning Improved and Efficient Networks
• Filter Grafting for Deep Neural Networks
• SQuantizer: Simultaneous Learning for Both Sparse and Low-precision Neural Networks
• CUP: Cluster Pruning for Compressing Deep Neural Networks
• Espresso: Efficient Forward Propagation for BCNNs
• PENNI: Pruned Kernel Sharing for Efficient CNN Inference
• Scalable Model Compression by Entropy Penalized Reparameterization
• Campfire: Compressible, Regularization-Free, Structured Sparse Training for Hardware Accelerators
• S3NAS: Fast NPU-aware Neural Architecture Search Methodology
• Enabling Retrain-free Deep Neural Network Pruning using Surrogate Lagrangian Relaxation
• Fast Neural Architecture Construction using EnvelopeNets
• BrainSlug: Transparent Acceleration of Deep Learning Through Depth-First Parallelism
• Intermittent Inference with Nonuniformly Compressed Multi-Exit Neural Network for Energy Harvesting Powered Devices
• Hu-Fu: Hardware and Software Collaborative Attack Framework against Neural Networks
• Exploring Gradient Flow Based Saliency for DNN Model Compression
• Cascaded channel pruning using hierarchical self-distillation
• Auxiliary Learning for Deep Multi-task Learning
• Convolutional Neural Network Simplification with Progressive Retraining
• Adaptive Dense-to-Sparse Paradigm for Pruning Online Recommendation System with Non-Stationary Data
• COP: Customized Deep Model Compression via Regularized Correlation-Based Filter-Level Pruning
• Unifying and Merging Well-trained Deep Neural Networks for Inference Stage
• CDFI: Compression-Driven Network Design for Frame Interpolation
• AutoGrow: Automatic Layer Growing in Deep Convolutional Networks
• Neural Network Compression Via Sparse Optimization
• An Image Enhancing Pattern-based Sparsity for Real-time Inference on Mobile Devices
• Automatic Neural Network Compression by Sparsity-Quantization Joint Learning: A Constrained Optimization-based Approach
• BitNet: Bit-Regularized Deep Neural Networks
• Personalized Federated Learning by Structured and Unstructured Pruning under Data Heterogeneity
• Pufferfish: Communication-efficient Models At No Extra Cost
• High Performance Convolution Using Sparsity and Patterns for Inference in Deep Convolutional Neural Networks
• Direction is what you need: Improving Word Embedding Compression in Large Language Models
• DARC: Differentiable ARchitecture Compression
• FlexSA: Flexible Systolic Array Architecture for Efficient Pruned DNN Model Training
• Efficient Hardware Realization of Convolutional Neural Networks using Intra-Kernel Regular Pruning
• Cogradient Descent for Bilinear Optimization
• Hessian-Aware Pruning and Optimal Neural Implant
• Dynamic Group Convolution for Accelerating Convolutional Neural Networks
• Dynamic Sparse Graph for Efficient Deep Learning
• How Not to Give a FLOP: Combining Regularization and Pruning for Efficient Inference
• REPrune: Filter Pruning via Representative Election
• VACL: Variance-Aware Cross-Layer Regularization for Pruning Deep Residual Networks
• DeepLight: Deep Lightweight Feature Interactions for Accelerating CTR Predictions in Ad Serving
• RSO: A Gradient Free Sampling Based Approach For Training Deep Neural Networks
• A High-Performance Adaptive Quantization Approach for Edge CNN Applications
• MutualNet: Adaptive ConvNet via Mutual Learning from Network Width and Resolution
• Sparse Flows: Pruning Continuous-depth Models
• Towards Latency-aware DNN Optimization with GPU Runtime Analysis and Tail Effect Elimination
• WAFFLE: Watermarking in Federated Learning
• A Generalized and Robust Method Towards Practical Gaze Estimation on Smart Phone
• Single-shot Channel Pruning Based on Alternating Direction Method of Multipliers
• Data-Free Knowledge Distillation with Soft Targeted Transfer Set Synthesis
• Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition
• A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustness
• Nonconvex sparse regularization for deep neural networks and its optimality
• ACDC: Weight Sharing in Atom-Coefficient Decomposed Convolution
• Enabling On-Device CNN Training by Self-Supervised Instance Filtering and Error Map Pruning
• AACP: Model Compression by Accurate and Automatic Channel Pruning
• Network Automatic Pruning: Start NAP and Take a Nap
• Efficient CNN-LSTM based Image Captioning using Neural Network Compression
• Advances in the training, pruning and enforcement of shape constraints of Morphological Neural Networks using Tropical Algebra
• Deep Neural Compression Via Concurrent Pruning and Self-Distillation
• DS-Net++: Dynamic Weight Slicing for Efficient Inference in CNNs and Transformers
• AdaptCL: Efficient Collaborative Learning with Dynamic and Adaptive Pruning
• Knowledge Distillation via Instance-level Sequence Learning
• Joint-DetNAS: Upgrade Your Detector with NAS, Pruning and Dynamic Distillation
• Neural network is heterogeneous: Phase matters more
• Pruning Filter in Filter
• SASL: Saliency-Adaptive Sparsity Learning for Neural Network Acceleration
• Deleting object selective units in a fully-connected layer of deep convolutional networks improves classification performance
• Single-Net Continual Learning with Progressive Segmented Training (PST)
• ECC: Platform-Independent Energy-Constrained Deep Neural Network Compression via a Bilinear Regression Model
• Efficient Neural Network Compression
• LeanResNet: A Low-cost Yet Effective Convolutional Residual Networks
• HENet:A Highly Efficient Convolutional Neural Networks Optimized for Accuracy, Speed and Storage
• Fast DCTTS: Efficient Deep Convolutional Text-to-Speech
• Extending Label Smoothing Regularization with Self-Knowledge Distillation
• ThumbNet: One Thumbnail Image Contains All You Need for Recognition
• Learning from a Lightweight Teacher for Efficient Knowledge Distillation
• AdaPruner: Adaptive Channel Pruning and Effective Weights Inheritance
• On the Orthogonality of Knowledge Distillation with Other Techniques: From an Ensemble Perspective
• Filter Pruning using Hierarchical Group Sparse Regularization for Deep Convolutional Neural Networks
• DECORE: Deep Compression with Reinforcement Learning
• Training Sparse Neural Networks using Compressed Sensing
• Deep Learning in the Era of Edge Computing: Challenges and Opportunities
• UCP: Uniform Channel Pruning for Deep Convolutional Neural Networks Compression and Acceleration
• Efficient Crowd Counting via Structured Knowledge Transfer
• Dissecting Pruned Neural Networks
• JavaScript Convolutional Neural Networks for Keyword Spotting in the Browser: An Experimental Analysis
• Neuron ranking -- an informed way to condense convolutional neural networks architecture
• Paying more attention to snapshots of Iterative Pruning: Improving Model Compression via Ensemble Distillation
• Ex uno plures: Splitting One Model into an Ensemble of Subnetworks
• Localization-aware Channel Pruning for Object Detection
• A Feature-map Discriminant Perspective for Pruning Deep Neural Networks
• Complexity-aware Adaptive Training and Inference for Edge-Cloud Distributed AI Systems
• ResIST: Layer-Wise Decomposition of ResNets for Distributed Training
• Pruning Ternary Quantization
• Gator: Customizable Channel Pruning of Neural Networks with Gating
• TOCO: A Framework for Compressing Neural Network Models Based on Tolerance Analysis
• Efficient Micro-Structured Weight Unification and Pruning for Neural Network Compression
• FATNN: Fast and Accurate Ternary Neural Networks
• Cross-Channel Intragroup Sparsity Neural Network
• Finding Everything within Random Binary Networks
• AutoPruning for Deep Neural Network with Dynamic Channel Masking
• Class-dependent Compression of Deep Neural Networks
• Splitting Convolutional Neural Network Structures for Efficient Inference
• Digging Deeper into CRNN Model in Chinese Text Images Recognition
• An Inter-Layer Weight Prediction and Quantization for Deep Neural Networks based on a Smoothly Varying Weight Hypothesis
• The Role of Regularization in Shaping Weight and Node Pruning Dependency and Dynamics
• Diversifying Inference Path Selection: Moving-Mobile-Network for Landmark Recognition
• Self-Reorganizing and Rejuvenating CNNs for Increasing Model Capacity Utilization
• Towards thinner convolutional neural networks through Gradually Global Pruning
• Studying the Plasticity in Deep Convolutional Neural Networks using Random Pruning
• Dynamic Slimmable Network
• Channel-wise pruning of neural networks with tapering resource constraint
• Neural Rejuvenation: Improving Deep Network Training by Enhancing Computational Resource Utilization
• Grassmannian Packings in Neural Networks: Learning with Maximal Subspace Packings for Diversity and Anti-Sparsity
• Efficient architecture for deep neural networks with heterogeneous sensitivity
• How Do You Act? An Empirical Study to Understand Behavior of Deep Reinforcement Learning Agents
• ASCAI: Adaptive Sampling for acquiring Compact AI
• Towards Efficient Neural Networks On-a-chip: Joint Hardware-Algorithm Approaches
• Sparse Mixture of Local Experts for Efficient Speech Enhancement
• Quantization Mimic: Towards Very Tiny CNN for Object Detection
• Sparse Persistent RNNs: Squeezing Large Recurrent Networks On-Chip
• Differentiable Feature Aggregation Search for Knowledge Distillation
• Improving Network Slimming with Nonconvex Regularization
• An Efficient Quantitative Approach for Optimizing Convolutional Neural Networks
• DNN Quantization with Attention
• Speeding up convolutional networks pruning with coarse ranking
• Condensing Two-stage Detection with Automatic Object Key Part Discovery
• Structured Ensembles: an Approach to Reduce the Memory Footprint of Ensemble Methods
• Cogradient Descent for Dependable Learning
• Network Pruning via Annealing and Direct Sparsity Control
• AsymmNet: Towards ultralight convolution neural networks using asymmetrical bottlenecks
• Simultaneously Learning Architectures and Features of Deep Neural Networks
• Network Adjustment: Channel Search Guided by FLOPs Utilization Ratio
• Self-grouping Convolutional Neural Networks
• Toward Compact Parameter Representations for Architecture-Agnostic Neural Network Compression
• Penetrating the Fog: the Path to Efficient CNN Models
• Neural networks adapting to datasets: learning network size and topology
• MoEVC: A Mixture-of-experts Voice Conversion System with Sparse Gating Mechanism for Accelerating Online Computation
• Investigating Channel Pruning through Structural Redundancy Reduction -- A Statistical Study
• Directed-Weighting Group Lasso for Eltwise Blocked CNN Pruning
• Automatic low-bit hybrid quantization of neural networks through meta learning
• Auto Deep Compression by Reinforcement Learning Based Actor-Critic Structure
• AUTOKD: Automatic Knowledge Distillation Into A Student Architecture Family
• AntiDote: Attention-based Dynamic Optimization for Neural Network Runtime Efficiency
• Channel Planting for Deep Neural Networks using Knowledge Distillation
• Online Filter Clustering and Pruning for Efficient Convnets
• Class-Discriminative CNN Compression
• Activation Map Adaptation for Effective Knowledge Distillation
• GHFP: Gradually Hard Filter Pruning
• Neural Plasticity Networks
• Grow-Push-Prune: aligning deep discriminants for effective structural network compression
• An Improving Framework of regularization for Network Compression
• SdcNet: A Computation-Efficient CNN for Object Recognition
• Feature Statistics Guided Efficient Filter Pruning
• Light Multi-segment Activation for Model Compression
• Multi-Task Network Pruning and Embedded Optimization for Real-time Deployment in ADAS
• Width Transfer: On the (In)variance of Width Optimization
• HALO: Learning to Prune Neural Networks with Shrinkage
• SVMax: A Feature Embedding Regularizer
• Associative Convolutional Layers
• HOTCAKE: Higher Order Tucker Articulated Kernels for Deeper CNN Compression
• Student Helping Teacher: Teacher Evolution via Self-Knowledge Distillation
• A Note on Latency Variability of Deep Neural Networks for Mobile Inference
• EZCrop: Energy-Zoned Channels for Robust Output Pruning
• Optimising Resource Management for Embedded Machine Learning
• Blending Pruning Criteria for Convolutional Neural Networks