Stabilizing the Lottery Ticket Hypothesis

Abstract

Pruning is a well-established technique for removing unnecessary structure from neural networks after training to improve the performance of inference. Several recent results have explored the possibility of pruning at initialization time to provide similar benefits during training. In particular, the "lottery ticket hypothesis" conjectures that typical neural networks contain small subnetworks that can train to similar accuracy in a commensurate number of steps. The evidence for this claim is that a procedure based on iterative magnitude pruning (IMP) reliably finds such subnetworks retroactively on small vision tasks. However, IMP fails on deeper networks, and proposed methods to prune before training or train pruned networks encounter similar scaling limitations. In this paper, we argue that these efforts have struggled on deeper networks because they have focused on pruning precisely at initialization. We modify IMP to search for subnetworks that could have been obtained by pruning early in training (0.1% to 7% through) rather than at iteration 0. With this change, it finds small subnetworks of deeper networks (e.g., 80% sparsity on Resnet-50) that can complete the training process to match the accuracy of the original network on more challenging tasks (e.g., ImageNet). In situations where IMP fails at iteration 0, the accuracy benefits of delaying pruning accrue rapidly over the earliest iterations of training. To explain these behaviors, we study subnetwork "stability," finding that - as accuracy improves in this fashion - IMP subnetworks train to parameters closer to those of the full network and do so with improved consistency in the face of gradient noise. These results offer new insights into the opportunity to prune large-scale networks early in training and the behaviors underlying the lottery ticket hypothesis

This article has been subsumed by "Linear Mode Connectivity and the Lottery Ticket Hypothesis" (arXiv:1912.05671, ICML 2020). Please read/cite that article instead

References in corpus (15)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Rethinking the Value of Network Pruning
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• A Convergence Theory for Deep Learning via Over-Parameterization
• Pruning Filters for Efficient ConvNets
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science
• The State of Sparsity in Deep Neural Networks
• Deep Rewiring: Training very sparse deep networks
• Uniform convergence may be unable to explain generalization in deep learning
• Inductive Bias of Deep Convolutional Networks through Pooling Geometry

Cited by in corpus (62)

• Optimization for deep learning: theory and algorithms
• Dynamic Model Pruning with Feedback
• Chasing Sparsity in Vision Transformers: An End-to-End Exploration
• Pruning neural networks without any data by iteratively conserving synaptic flow
• Optimizing Speech Recognition For The Edge
• The Early Phase of Neural Network Training
• Winning the Lottery with Continuous Sparsification
• Only Train Once: A One-Shot Neural Network Training And Pruning Framework
• Progressive Skeletonization: Trimming more fat from a network at initialization
• The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
• Rethinking Weight Decay For Efficient Neural Network Pruning
• Learning Sparse Sharing Architectures for Multiple Tasks
• FreezeNet: Full Performance by Reduced Storage Costs
• Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimization
• Efficient Neural Network Training via Forward and Backward Propagation Sparsification
• The Search for Sparse, Robust Neural Networks
• Luck Matters: Understanding Training Dynamics of Deep ReLU Networks
• Self-Damaging Contrastive Learning
• Pruning via Iterative Ranking of Sensitivity Statistics
• Sparse Training via Boosting Pruning Plasticity with Neuroregeneration
• Directional Pruning of Deep Neural Networks
• AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networks
• TIRAMISU: A Polyhedral Compiler for Dense and Sparse Deep Learning
• Lottery Tickets in Linear Models: An Analysis of Iterative Magnitude Pruning
• Towards Practical Lottery Ticket Hypothesis for Adversarial Training
• Learning to Prune Deep Neural Networks via Reinforcement Learning
• Powerpropagation: A sparsity inducing weight reparameterisation
• Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models
• Revisiting Loss Modelling for Unstructured Pruning
• The Differentially Private Lottery Ticket Mechanism
• Diet deep generative audio models with structured lottery
• Activation Density driven Energy-Efficient Pruning in Training
• ESPN: Extremely Sparse Pruned Networks
• Learning Compact Representations of Neural Networks using DiscriminAtive Masking (DAM)
• Bespoke vs. Prêt-à-Porter Lottery Tickets: Exploiting Mask Similarity for Trainable Sub-Network Finding
• How many winning tickets are there in one DNN?
• Fixing the Teacher-Student Knowledge Discrepancy in Distillation
• Learning to Act through Evolution of Neural Diversity in Random Neural Networks
• Dissecting Pruned Neural Networks
• Why Lottery Ticket Wins? A Theoretical Perspective of Sample Complexity on Pruned Neural Networks
• How many degrees of freedom do we need to train deep networks: a loss landscape perspective
• MBGD-RDA Training and Rule Pruning for Concise TSK Fuzzy Regression Models
• Dynamic Sparsity Neural Networks for Automatic Speech Recognition
• Troubleshooting Blind Image Quality Models in the Wild
• Student Specialization in Deep ReLU Networks With Finite Width and Input Dimension
• Ultra-light deep MIR by trimming lottery tickets
• Juvenile state hypothesis: What we can learn from lottery ticket hypothesis researches?
• Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization
• Search Spaces for Neural Model Training
• On Lottery Tickets and Minimal Task Representations in Deep Reinforcement Learning
• The Elastic Lottery Ticket Hypothesis
• Spending Your Winning Lottery Better After Drawing It
• Sparse Perturbations for Improved Convergence in Stochastic Zeroth-Order Optimization
• Modulating Regularization Frequency for Efficient Compression-Aware Model Training
• Sifting out the features by pruning: Are convolutional networks the winning lottery ticket of fully connected ones?
• Masked Training of Neural Networks with Partial Gradients
• How Well Do Sparse Imagenet Models Transfer?
• Learning the mapping $\mathbf{x}\mapsto {\sum }_{i=1}^d{x}_i^2$: the cost of finding the needle in a haystack
• Putting 3D Spatially Sparse Networks on a Diet
• Is Feature Diversity Necessary in Neural Network Initialization?
• Membership Inference Attacks on Lottery Ticket Networks
• Efficient Sparse Artificial Neural Networks