Learned Threshold Pruning

Abstract

This paper presents a novel differentiable method for unstructured weight pruning of deep neural networks. Our learned-threshold pruning (LTP) method learns per-layer thresholds via gradient descent, unlike conventional methods where they are set as input. Making thresholds trainable also makes LTP computationally efficient, hence scalable to deeper networks. For example, it takes $30$ epochs for LTP to prune ResNet50 on ImageNet by a factor of $9.1$. This is in contrast to other methods that search for per-layer thresholds via a computationally intensive iterative pruning and fine-tuning process. Additionally, with a novel differentiable $L_0$ regularization, LTP is able to operate effectively on architectures with batch-normalization. This is important since $L_1$ and $L_2$ penalties lose their regularizing effect in networks with batch-normalization. Finally, LTP generates a trail of progressively sparser networks from which the desired pruned network can be picked based on sparsity and performance requirements. These features allow LTP to achieve competitive compression rates on ImageNet networks such as AlexNet ($26.4\times$ compression with $79.1\%$ Top-5 accuracy) and ResNet50 ($9.1\times$ compression with $92.0\%$ Top-5 accuracy). We also show that LTP effectively prunes modern \textit{compact} architectures, such as EfficientNet, MobileNetV2 and MixNet.

References in corpus (21)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Fixed Point Quantization of Deep Convolutional Networks
• A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Bayesian Compression for Deep Learning
• Variational Dropout Sparsifies Deep Neural Networks
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• MixConv: Mixed Depthwise Convolutional Kernels
• L2 Regularization versus Batch and Weight Normalization
• Compressing Neural Networks using the Variational Information Bottleneck
• Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications
• Soft Threshold Weight Reparameterization for Learnable Sparsity
• Norm matters: efficient and accurate normalization schemes in deep networks
• Soft Weight-Sharing for Neural Network Compression
• Automated Pruning for Deep Neural Network Compression
• Progressive DNN Compression: A Key to Achieve Ultra-High Weight Pruning and Quantization Rates using ADMM
• Taxonomy and Evaluation of Structured Compression of Convolutional Neural Networks

Cited by in corpus (6)

• Soft Threshold Weight Reparameterization for Learnable Sparsity
• DASS: Differentiable Architecture Search for Sparse neural networks
• Structured Convolutions for Efficient Neural Network Design
• An Expectation-Maximization Perspective on Federated Learning
• Learning Pruned Structure and Weights Simultaneously from Scratch: an Attention based Approach
• HALO: Learning to Prune Neural Networks with Shrinkage