Data-Independent Structured Pruning of Neural Networks via Coresets
arXiv:2008.08316
Abstract
Model compression is crucial for deployment of neural networks on devices with limited computational and memory resources. Many different methods show comparable accuracy of the compressed model and similar compression rates. However, the majority of the compression methods are based on heuristics and offer no worst-case guarantees on the trade-off between the compression rate and the approximation error for an arbitrarily new sample. We propose the first efficient structured pruning algorithm with a provable trade-off between its compression rate and the approximation error for any future test sample. Our method is based on the coreset framework and it approximates the output of a layer of neurons/filters by a coreset of neurons/filters in the previous layer and discards the rest. We apply this framework in a layer-by-layer fashion from the bottom to the top. Unlike previous works, our coreset is data independent, meaning that it provably guarantees the accuracy of the function for any input , including an adversarial one.
References in corpus (10)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
- AMC: AutoML for Model Compression and Acceleration on Mobile Devices
- Rethinking the Value of Network Pruning
- Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
- The State of Sparsity in Deep Neural Networks
- Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
- Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks
- A Note on Randomized Element-wise Matrix Sparsification