Structured Pruning for Deep Convolutional Neural Networks: A survey

Abstract

The remarkable performance of deep Convolutional neural networks (CNNs) is generally attributed to their deeper and wider architectures, which can come with significant computational costs. Pruning neural networks has thus gained interest since it effectively lowers storage and computational costs. In contrast to weight pruning, which results in unstructured models, structured pruning provides the benefit of realistic acceleration by producing models that are friendly to hardware implementation. The special requirements of structured pruning have led to the discovery of numerous new challenges and the development of innovative solutions. This article surveys the recent progress towards structured pruning of deep CNNs. We summarize and compare the state-of-the-art structured pruning techniques with respect to filter ranking methods, regularization methods, dynamic execution, neural architecture search, the lottery ticket hypothesis, and the applications of pruning. While discussing structured pruning algorithms, we briefly introduce the unstructured pruning counterpart to emphasize their differences. Furthermore, we provide insights into potential research opportunities in the field of structured pruning. A curated list of neural network pruning papers can be found at https://github.com/he-y/Awesome-Pruning . A dedicated website offering a more interactive comparison of structured pruning methods can be found at: https://huggingface.co/spaces/he-yang/Structured-Pruning-Survey .

arXiv commentAccepted by IEEE Transactions on Pattern Analysis and Machine Intelligence

References in corpus (21)

• Representation Learning with Contrastive Predictive Coding
• Knowledge Distillation: A Survey
• On the Opportunities and Risks of Foundation Models
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks
• ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better
• Binary Neural Networks: A Survey
• Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks
• Hardware and Software Optimizations for Accelerating Deep Neural Networks: Survey of Current Trends, Challenges, and the Road Ahead
• A Survey on Methods and Theories of Quantized Neural Networks
• An Entropy-based Pruning Method for CNN Compression
• Bayesian Bits: Unifying Quantization and Pruning
• Filter Pruning by Switching to Neighboring CNNs with Good Attributes
• EDropout: Energy-Based Dropout and Pruning of Deep Neural Networks
• Overcoming Long-term Catastrophic Forgetting through Adversarial Neural Pruning and Synaptic Consolidation
• DAIS: Automatic Channel Pruning via Differentiable Annealing Indicator Search
• Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Survey
• Joint Multi-Dimension Pruning via Numerical Gradient Update

Cited by in corpus (8)

• Vision Transformers on the Edge: A Comprehensive Survey of Model Compression and Acceleration Strategies
• EEG Emotion Copilot: Optimizing Lightweight LLMs for Emotional EEG Interpretation with Assisted Medical Record Generation
• Hardware-Aware DNN Compression for Homogeneous Edge Devices
• Hardware-Aware DNN Compression for Homogeneous Edge Devices
• Layer Pruning with Consensus: A Triple-Win Solution
• Polynomially Over-Parameterized Convolutional Neural Networks Contain Structured Strong Winning Lottery Tickets
• Energy-Aware LLMs: A step towards sustainable AI for downstream applications
• Robust Visual Localization in Compute-Constrained Environments by Salient Edge Rendering and Weighted Hamming Similarity