Winning the Lottery with Continuous Sparsification
arXiv:1912.04427
Abstract
The search for efficient, sparse deep neural network models is most prominently performed by pruning: training a dense, overparameterized network and removing parameters, usually via following a manually-crafted heuristic. Additionally, the recent Lottery Ticket Hypothesis conjectures that, for a typically-sized neural network, it is possible to find small sub-networks which, when trained from scratch on a comparable budget, match the performance of the original dense counterpart. We revisit fundamental aspects of pruning algorithms, pointing out missing ingredients in previous approaches, and develop a method, Continuous Sparsification, which searches for sparse networks based on a novel approximation of an intractable regularization. We compare against dominant heuristic-based methods on pruning as well as ticket search -- finding sparse subnetworks that can be successfully re-trained from an early iterate. Empirical results show that we surpass the state-of-the-art for both objectives, across models and datasets, including VGG trained on CIFAR-10 and ResNet-50 trained on ImageNet. In addition to setting a new standard for pruning, Continuous Sparsification also offers fast parallel ticket search, opening doors to new applications of the Lottery Ticket Hypothesis.
arXiv commentPublished as a conference paper at NeurIPS 2020
References in corpus (17)
- SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
- Categorical Reparameterization with Gumbel-Softmax
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- DARTS: Differentiable Architecture Search
- AMC: AutoML for Model Compression and Acceleration on Mobile Devices
- A Convergence Theory for Deep Learning via Over-Parameterization
- To prune, or not to prune: exploring the efficacy of pruning for model compression
- The State of Sparsity in Deep Neural Networks
- Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
- Sparse Networks from Scratch: Faster Training without Losing Performance
- Stabilizing the Lottery Ticket Hypothesis
- Rigging the Lottery: Making All Tickets Winners
- Soft Threshold Weight Reparameterization for Learnable Sparsity
- Discovering Neural Wirings
- Learning Implicitly Recurrent CNNs Through Parameter Sharing
- Sparse Transfer Learning via Winning Lottery Tickets
- Training Sparse Neural Networks
Cited by in corpus (17)
- Soft Threshold Weight Reparameterization for Learnable Sparsity
- Linear Mode Connectivity and the Lottery Ticket Hypothesis
- A Unified Lottery Ticket Hypothesis for Graph Neural Networks
- A Brain-inspired Algorithm for Training Highly Sparse Neural Networks
- Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training
- Dynamic Sparse Learning: A Novel Paradigm for Efficient Recommendation
- Efficient Neural Network Training via Forward and Backward Propagation Sparsification
- Sparse Training via Boosting Pruning Plasticity with Neuroregeneration
- Adapting by Pruning: A Case Study on BERT
- Growing Efficient Deep Networks by Structured Continuous Sparsification
- Data-Efficient GAN Training Beyond (Just) Augmentations: A Lottery Ticket Perspective
- Drawing Robust Scratch Tickets: Subnetworks with Inborn Robustness Are Found within Randomly Initialized Networks
- ESPN: Extremely Sparse Pruned Networks
- Spending Your Winning Lottery Better After Drawing It
- HALO: Learning to Prune Neural Networks with Shrinkage
- On the Existence of Universal Lottery Tickets
- Plant 'n' Seek: Can You Find the Winning Ticket?