Joint Pruning & Quantization for Extremely Sparse Neural Networks
arXiv:2010.01892
Abstract
We investigate pruning and quantization for deep neural networks. Our goal is to achieve extremely high sparsity for quantized networks to enable implementation on low cost and low power accelerator hardware. In a practical scenario, there are particularly many applications for dense prediction tasks, hence we choose stereo depth estimation as target. We propose a two stage pruning and quantization pipeline and introduce a Taylor Score alongside a new fine-tuning mode to achieve extreme sparsity without sacrificing performance. Our evaluation does not only show that pruning and quantization should be investigated jointly, but also shows that almost 99% of memory demand can be cut while hardware costs can be reduced up to 99.9%. In addition, to compare with other works, we demonstrate that our pruning stage alone beats the state-of-the-art when applied to ResNet on CIFAR10 and ImageNet.
13 page, 16 figures
References in corpus (13)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Deep Residual Learning for Image Recognition
- Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
- Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
- BinaryConnect: Training Deep Neural Networks with binary weights during propagations
- Ternary Weight Networks
- Pruning Filters for Efficient ConvNets
- Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
- Pyramid Stereo Matching Network
- Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration
- Towards Effective Low-bitwidth Convolutional Neural Networks
- DMCP: Differentiable Markov Channel Pruning for Neural Networks
- Neural Network Pruning with Residual-Connections and Limited-Data