A Probabilistic Approach to Neural Network Pruning
arXiv:2105.10065
Abstract
Neural network pruning techniques reduce the number of parameters without compromising predicting ability of a network. Many algorithms have been developed for pruning both over-parameterized fully-connected networks (FCNs) and convolutional neural networks (CNNs), but analytical studies of capabilities and compression ratios of such pruned sub-networks are lacking. We theoretically study the performance of two pruning techniques (random and magnitude-based) on FCNs and CNNs. Given a target network {whose weights are independently sampled from appropriate distributions}, we provide a universal approach to bound the gap between a pruned and the target network in a probabilistic sense. The results establish that there exist pruned networks with expressive power within any specified bound from the target network.
References in corpus (11)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- A Survey of Model Compression and Acceleration for Deep Neural Networks
- Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
- Optimization for deep learning: theory and algorithms
- Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approach
- Proving the Lottery Ticket Hypothesis: Pruning is All You Need
- A Selective Overview of Deep Learning
- SiPPing Neural Networks: Sensitivity-informed Provable Pruning of Neural Networks
- Lottery Tickets in Linear Models: An Analysis of Iterative Magnitude Pruning
- Greedy Optimization Provably Wins the Lottery: Logarithmic Number of Winning Tickets is Enough