What is the State of Neural Network Pruning?

Abstract

Neural network pruning---the task of reducing the size of a network by removing parameters---has been the subject of a great deal of work in recent years. We provide a meta-analysis of the literature, including an overview of approaches to pruning and consistent findings in the literature. After aggregating results across 81 papers and pruning hundreds of models in controlled conditions, our clearest finding is that the community suffers from a lack of standardized benchmarks and metrics. This deficiency is substantial enough that it is hard to compare pruning techniques to one another or determine how much progress the field has made over the past three decades. To address this situation, we identify issues with current practices, suggest concrete remedies, and introduce ShrinkBench, an open-source framework to facilitate standardized evaluations of pruning methods. We use ShrinkBench to compare various pruning techniques and show that its comprehensive evaluation can prevent common pitfalls when comparing pruning methods.

arXiv commentPublished in Proceedings of Machine Learning and Systems 2020 (MLSys 2020)

References in corpus (14)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• Rethinking the Value of Network Pruning
• Benchmark Analysis of Representative Deep Neural Network Architectures
• Bayesian Compression for Deep Learning
• Variational Dropout Sparsifies Deep Neural Networks
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• Sprintz: Time Series Compression for the Internet of Things
• PerforatedCNNs: Acceleration through Elimination of Redundant Convolutions
• The Power of Sparsity in Convolutional Neural Networks
• Balanced Sparsity for Efficient DNN Inference on GPU
• Diversity Networks: Neural Network Compression Using Determinantal Point Processes
• A General SIMD-based Approach to Accelerating Compression Algorithms
• Jointly Sparse Convolutional Neural Networks in Dual Spatial-Winograd Domains

Cited by in corpus (35)

• Bringing AI To Edge: From Deep Learning's Perspective
• What Do Compressed Deep Neural Networks Forget?
• Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
• Pruning neural networks without any data by iteratively conserving synaptic flow
• A Survey on Deep Neural Network Compression: Challenges, Overview, and Solutions
• Characterising Bias in Compressed Models
• WoodFisher: Efficient Second-Order Approximation for Neural Network Compression
• Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot
• The Feasibility and Inevitability of Stealth Attacks
• An Information Theory-inspired Strategy for Automatic Network Pruning
• Adapting by Pruning: A Case Study on BERT
• Lottery Tickets in Linear Models: An Analysis of Iterative Magnitude Pruning
• Deep Serial Number: Computational Watermarking for DNN Intellectual Property Protection
• Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings
• Revisiting Loss Modelling for Unstructured Pruning
• Adaptive Dense-to-Sparse Paradigm for Pruning Online Recommendation System with Non-Stationary Data
• Evolving Search Space for Neural Architecture Search
• Revisiting Deep Architectures for Head Motion Prediction in 360° Videos
• Stabilizing Elastic Weight Consolidation method in practical ML tasks and using weight importances for neural network pruning
• Troubleshooting Blind Image Quality Models in the Wild
• Understanding Learning Dynamics of Binary Neural Networks via Information Bottleneck
• Towards Efficient Convolutional Network Models with Filter Distribution Templates
• Towards Inference Delivery Networks: Distributing Machine Learning with Optimality Guarantees
• Gator: Customizable Channel Pruning of Neural Networks with Gating
• Why is Pruning at Initialization Immune to Reinitializing and Shuffling?
• Adaptive Distillation: Aggregating Knowledge from Multiple Paths for Efficient Distillation
• Dense for the Price of Sparse: Improved Performance of Sparsely Initialized Networks via a Subspace Offset
• Neural networks adapting to datasets: learning network size and topology
• Sifting out the features by pruning: Are convolutional networks the winning lottery ticket of fully connected ones?
• Against Membership Inference Attack: Pruning is All You Need
• Multi-Task Network Pruning and Embedded Optimization for Real-time Deployment in ADAS
• Network Compression for Machine-Learnt Fluid Simulations
• Place recognition survey: An update on deep learning approaches
• Toward Compact Parameter Representations for Architecture-Agnostic Neural Network Compression
• Robust error bounds for quantised and pruned neural networks