SiPPing Neural Networks: Sensitivity-informed Provable Pruning of Neural Networks

Abstract

We introduce a pruning algorithm that provably sparsifies the parameters of a trained model in a way that approximately preserves the model's predictive accuracy. Our algorithm uses a small batch of input points to construct a data-informed importance sampling distribution over the network's parameters, and adaptively mixes a sampling-based and deterministic pruning procedure to discard redundant weights. Our pruning method is simultaneously computationally efficient, provably accurate, and broadly applicable to various network architectures and data distributions. Our empirical comparisons show that our algorithm reliably generates highly compressed networks that incur minimal loss in performance relative to that of the original network. We present experimental results that demonstrate our algorithm's potential to unearth essential network connections that can be trained successfully in isolation, which may be of independent interest.

arXiv commentFirst two authors contributed equally

References in corpus (15)

• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Wide Residual Networks
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• The State of Sparsity in Deep Neural Networks
• Convolutional neural networks with low-rank regularization
• A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
• Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
• Compression-aware Training of Deep Networks
• Structured Transforms for Small-Footprint Deep Learning
• Dynamic Model Pruning with Feedback
• Binary embeddings with structured hashed projections
• Campfire: Compressible, Regularization-Free, Structured Sparse Training for Hardware Accelerators

Cited by in corpus (9)

• Good Subnetworks Provably Exist: Pruning via Greedy Forward Selection
• Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracy
• Provable Filter Pruning for Efficient Neural Networks
• Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition
• Sparse Flows: Pruning Continuous-depth Models
• A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustness
• Greedy Optimization Provably Wins the Lottery: Logarithmic Number of Winning Tickets is Enough
• Why Lottery Ticket Wins? A Theoretical Perspective of Sample Complexity on Pruned Neural Networks
• A Probabilistic Approach to Neural Network Pruning