Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask

Abstract

The recent "Lottery Ticket Hypothesis" paper by Frankle & Carbin showed that a simple approach to creating sparse networks (keeping the large weights) results in models that are trainable from scratch, but only when starting from the same initial weights. The performance of these networks often exceeds the performance of the non-sparse base model, but for reasons that were not well understood. In this paper we study the three critical components of the Lottery Ticket (LT) algorithm, showing that each may be varied significantly without impacting the overall results. Ablating these factors leads to new insights for why LT networks perform as well as they do. We show why setting weights to zero is important, how signs are all you need to make the reinitialized network train, and why masking behaves like training. Finally, we discover the existence of Supermasks, masks that can be applied to an untrained, randomly initialized network to produce a model with performance far better than chance (86% on MNIST, 41% on CIFAR-10).

arXiv commentNeurIPS 2019 camera ready version

References in corpus (4)

• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• Learning Structured Sparsity in Deep Neural Networks
• Bayesian Compression for Deep Learning

Cited by in corpus (49)

• Sparse Networks from Scratch: Faster Training without Losing Performance
• Optimization for deep learning: theory and algorithms
• Proving the Lottery Ticket Hypothesis: Pruning is All You Need
• The learnability scaling of quantum states: restricted Boltzmann machines
• The Early Phase of Neural Network Training
• Rethinking Weight Decay For Efficient Neural Network Pruning
• Optimal Lottery Tickets via SubsetSum: Logarithmic Over-Parameterization is Sufficient
• How fine can fine-tuning be? Learning efficient language models
• Geometric framework to predict structure from function in neural networks
• End-to-End Supermask Pruning: Learning to Prune Image Captioning Models
• Sparse Transfer Learning via Winning Lottery Tickets
• LCA: Loss Change Allocation for Neural Network Training
• Selfish Sparse RNN Training
• FreezeNet: Full Performance by Reduced Storage Costs
• DASS: Differentiable Architecture Search for Sparse neural networks
• Luck Matters: Understanding Training Dynamics of Deep ReLU Networks
• Evolving and Merging Hebbian Learning Rules: Increasing Generalization by Decreasing the Number of Rules
• Observation Space Matters: Benchmark and Optimization Algorithm
• Pruning via Iterative Ranking of Sensitivity Statistics
• Heavy Tails in SGD and Compressibility of Overparametrized Neural Networks
• Calibrate and Prune: Improving Reliability of Lottery Tickets Through Prediction Calibration
• Pruning Randomly Initialized Neural Networks with Iterative Randomization
• Topological Insights into Sparse Neural Networks
• On Iterative Neural Network Pruning, Reinitialization, and the Similarity of Masks
• Trainless Model Performance Estimation for Neural Architecture Search
• Image Captioning with Sparse Recurrent Neural Network
• How many winning tickets are there in one DNN?
• Slot Machines: Discovering Winning Combinations of Random Weights in Neural Networks
• ESPN: Extremely Sparse Pruned Networks
• Bespoke vs. Prêt-à-Porter Lottery Tickets: Exploiting Mask Similarity for Trainable Sub-Network Finding
• ReaLPrune: ReRAM Crossbar-aware Lottery Ticket Pruned CNNs
• WaLDORf: Wasteless Language-model Distillation On Reading-comprehension
• Effect of the initial configuration of weights on the training and function of artificial neural networks
• Deep network as memory space: complexity, generalization, disentangled representation and interpretability
• NeuralScale: Efficient Scaling of Neurons for Resource-Constrained Deep Neural Networks
• Mining the Weights Knowledge for Optimizing Neural Network Structures
• Training highly effective connectivities within neural networks with randomly initialized, fixed weights
• Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization
• Class-dependent Compression of Deep Neural Networks
• Finding Everything within Random Binary Networks
• The Elastic Lottery Ticket Hypothesis
• Efficient Sparse Artificial Neural Networks
• Out-of-the-box channel pruned networks
• Putting 3D Spatially Sparse Networks on a Diet
• Sifting out the features by pruning: Are convolutional networks the winning lottery ticket of fully connected ones?
• Against Membership Inference Attack: Pruning is All You Need
• HALO: Learning to Prune Neural Networks with Shrinkage
• Self-Constructing Neural Networks Through Random Mutation
• Learning Digital Circuits: A Journey Through Weight Invariant Self-Pruning Neural Networks