HALO: Learning to Prune Neural Networks with Shrinkage

Abstract

Deep neural networks achieve state-of-the-art performance in a variety of tasks by extracting a rich set of features from unstructured data, however this performance is closely tied to model size. Modern techniques for inducing sparsity and reducing model size are (1) network pruning, (2) training with a sparsity inducing penalty, and (3) training a binary mask jointly with the weights of the network. We study different sparsity inducing penalties from the perspective of Bayesian hierarchical models and present a novel penalty called Hierarchical Adaptive Lasso (HALO) which learns to adaptively sparsify weights of a given network via trainable parameters. When used to train over-parametrized networks, our penalty yields small subnetworks with high accuracy without fine-tuning. Empirically, on image recognition tasks, we find that HALO is able to learn highly sparse network (only 5% of the parameters) with significant gains in performance over state-of-the-art magnitude pruning methods at the same level of sparsity. Code is available at https://github.com/skyler120/sparsity-halo.

arXiv commentAccepted at SDM 2021

References in corpus (28)

• Nearly unbiased variable selection under minimax concave penalty
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Understanding deep learning requires rethinking generalization
• Rethinking the Value of Network Pruning
• Pruning Filters for Efficient ConvNets
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Sparsity information and regularization in the horseshoe and other shrinkage priors
• Channel Pruning for Accelerating Very Deep Neural Networks
• Learning Sparse Neural Networks through $L_0$ Regularization
• Exploring Sparsity in Recurrent Neural Networks
• Compression-aware Training of Deep Networks
• Memory Bounded Deep Convolutional Networks
• Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask
• Soft Threshold Weight Reparameterization for Learnable Sparsity
• Generalized Beta Mixtures of Gaussians
• Stronger generalization bounds for deep nets via a compression approach
• Stochastic Hyperparameter Optimization through Hypernetworks
• Proving the Lottery Ticket Hypothesis: Pruning is All You Need
• Statistical estimation and testing via the sorted L1 norm
• How Can We Be So Dense? The Benefits of Using Highly Sparse Representations
• Big Neural Networks Waste Capacity
• Winning the Lottery with Continuous Sparsification
• Learned Threshold Pruning
• Posterior Concentration for Sparse Deep Learning
• Comparisons of penalized least squares methods by simulations
• Bayesian Masking: Sparse Bayesian Estimation with Weaker Shrinkage Bias
• Learning Optimal Linear Regularizers