Soft Threshold Weight Reparameterization for Learnable Sparsity

Abstract

Sparsity in Deep Neural Networks (DNNs) is studied extensively with the focus of maximizing prediction accuracy given an overall parameter budget. Existing methods rely on uniform or heuristic non-uniform sparsity budgets which have sub-optimal layer-wise parameter allocation resulting in a) lower prediction accuracy or b) higher inference cost (FLOPs). This work proposes Soft Threshold Reparameterization (STR), a novel use of the soft-threshold operator on DNN weights. STR smoothly induces sparsity while learning pruning thresholds thereby obtaining a non-uniform sparsity budget. Our method achieves state-of-the-art accuracy for unstructured sparsity in CNNs (ResNet50 and MobileNetV1 on ImageNet-1K), and, additionally, learns non-uniform budgets that empirically reduce the FLOPs by up to 50%. Notably, STR boosts the accuracy over existing results by up to 10% in the ultra sparse (99%) regime and can also be used to induce low-rank (structured sparsity) in RNNs. In short, STR is a simple mechanism which learns effective sparsity budgets that contrast with popular heuristics. Code, pretrained models and sparsity budgets are at https://github.com/RAIVNLab/STR.

19 pages, 10 figures, Published at International Conference on Machine Learning (ICML) 2020

References in corpus (22)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
• Rethinking the Value of Network Pruning
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science
• The State of Sparsity in Deep Neural Networks
• Variational Dropout Sparsifies Deep Neural Networks
• Sparse Networks from Scratch: Faster Training without Losing Performance
• Deep Rewiring: Training very sparse deep networks
• AutoSlim: Towards One-Shot Architecture Search for Channel Numbers
• Exploring Sparsity in Recurrent Neural Networks
• Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization
• Dynamic Model Pruning with Feedback
• On Iterative Hard Thresholding Methods for High-dimensional M-Estimation
• Discovering Neural Wirings
• Winning the Lottery with Continuous Sparsification
• Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers
• Learned Threshold Pruning
• Differentiable Sparsification for Deep Neural Networks

Cited by in corpus (30)

• Rigging the Lottery: Making All Tickets Winners
• Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch
• Layer-adaptive sparsity for the Magnitude-based Pruning
• Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers
• Winning the Lottery with Continuous Sparsification
• Progressive Skeletonization: Trimming more fat from a network at initialization
• WoodFisher: Efficient Second-Order Approximation for Neural Network Compression
• Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracy
• RNNPool: Efficient Non-linear Pooling for RAM Constrained Inference
• Learned Threshold Pruning
• A Brain-inspired Algorithm for Training Highly Sparse Neural Networks
• Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training
• Network Pruning That Matters: A Case Study on Retraining Variants
• Efficient Neural Network Training via Forward and Backward Propagation Sparsification
• Sparse Training via Boosting Pruning Plasticity with Neuroregeneration
• AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networks
• Representation Sparsification with Hybrid Thresholding for Fast SPLADE-based Document Retrieval
• Sparse Weight Activation Training
• Structured Convolutions for Efficient Neural Network Design
• Adaptive Dense-to-Sparse Paradigm for Pruning Online Recommendation System with Non-Stationary Data
• Effective Sparsification of Neural Networks with Global Sparsity Constraint
• Lottery Jackpots Exist in Pre-trained Models
• M-FAC: Efficient Matrix-Free Approximations of Second-Order Information
• AdaptCL: Efficient Collaborative Learning with Dynamic and Adaptive Pruning
• Pruning Filter in Filter
• Adjoined Networks: A Training Paradigm with Applications to Network Compression
• Cascade Weight Shedding in Deep Neural Networks: Benefits and Pitfalls for Network Pruning
• Alternate Model Growth and Pruning for Efficient Training of Recommendation Systems
• HALO: Learning to Prune Neural Networks with Shrinkage
• How Well Do Sparse Imagenet Models Transfer?