Spending Your Winning Lottery Better After Drawing It

Abstract

Lottery Ticket Hypothesis (LTH) suggests that a dense neural network contains a sparse sub-network that can match the performance of the original dense network when trained in isolation from scratch. Most works retrain the sparse sub-network with the same training protocols as its dense network, such as initialization, architecture blocks, and training recipes. However, till now it is unclear that whether these training protocols are optimal for sparse networks. In this paper, we demonstrate that it is unnecessary for spare retraining to strictly inherit those properties from the dense network. Instead, by plugging in purposeful "tweaks" of the sparse subnetwork architecture or its training recipe, its retraining can be significantly improved than the default, especially at high sparsity levels. Combining all our proposed "tweaks" can yield the new state-of-the-art performance of LTH, and these modifications can be easily adapted to other sparse training algorithms in general. Specifically, we have achieved a significant and consistent performance gain of1.05% - 4.93% for ResNet18 on CIFAR-100 over vanilla-LTH. Moreover, our methods are shown to generalize across datasets (CIFAR10, CIFAR100, TinyImageNet) and architectures (Vgg16, ResNet-18/ResNet-34, MobileNet). All codes will be publicly available.

References in corpus (20)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Rethinking the Value of Network Pruning
• When Does Label Smoothing Help?
• Searching for Activation Functions
• The State of Sparsity in Deep Neural Networks
• Comparing Rewinding and Fine-tuning in Neural Network Pruning
• Picking Winning Tickets Before Training by Preserving Gradient Flow
• Stabilizing the Lottery Ticket Hypothesis
• Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP
• Winning the Lottery with Continuous Sparsification
• When BERT Plays the Lottery, All Tickets Are Winning
• The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
• A Unified Lottery Ticket Hypothesis for Graph Neural Networks
• Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study
• EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets
• Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimization
• GradInit: Learning to Initialize Neural Networks for Stable and Efficient Training
• GANs Can Play Lottery Tickets Too