Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition
arXiv:2107.11442
Abstract
We present a novel global compression framework for deep neural networks that automatically analyzes each layer to identify the optimal per-layer compression ratio, while simultaneously achieving the desired overall compression. Our algorithm hinges on the idea of compressing each convolutional (or fully-connected) layer by slicing its channels into multiple groups and decomposing each group via low-rank decomposition. At the core of our algorithm is the derivation of layer-wise error bounds from the Eckart Young Mirsky theorem. We then leverage these bounds to frame the compression problem as an optimization problem where we wish to minimize the maximum compression error across layers and propose an efficient algorithm towards a solution. Our experiments indicate that our method outperforms existing low-rank compression approaches across a wide range of networks and data sets. We believe that our results open up new avenues for future research into the global performance-size trade-offs of modern neural networks. Our code is available at https://github.com/lucaslie/torchprune.
arXiv commentNeurIPS 2021
References in corpus (24)
- Distilling the Knowledge in a Neural Network
- Rethinking Atrous Convolution for Semantic Image Segmentation
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Wide Residual Networks
- Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
- Pruning Convolutional Neural Networks for Resource Efficient Inference
- Rethinking the Value of Network Pruning
- Pruning Filters for Efficient ConvNets
- Tensorizing Neural Networks
- Compressing Neural Networks with the Hashing Trick
- Convolutional neural networks with low-rank regularization
- Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups
- Dynamic Channel Pruning: Feature Boosting and Suppression
- Discrimination-aware Channel Pruning for Deep Neural Networks
- Compression-aware Training of Deep Networks
- Ultimate tensorization: compressing convolutional and FC layers alike
- Dynamic Model Pruning with Feedback
- Dimensionality Reduction via Regression in Hyperspectral Imagery
- Approximated Oracle Filter Pruning for Destructive CNN Width Optimization
- WoodFisher: Efficient Second-Order Approximation for Neural Network Compression
- ChipNet: Budget-Aware Pruning with Heaviside Continuous Approximations
- SiPPing Neural Networks: Sensitivity-informed Provable Pruning of Neural Networks
- Compressed Deep Networks: Goodbye SVD, Hello Robust Low-Rank Approximation
- Sparse Flows: Pruning Continuous-depth Models