Picking Winning Tickets Before Training by Preserving Gradient Flow

Abstract

Overparameterization has been shown to benefit both the optimization and generalization of neural networks, but large networks are resource hungry at both training and test time. Network pruning can reduce test-time resource requirements, but is typically applied to trained networks and therefore cannot avoid the expensive training process. We aim to prune networks at initialization, thereby saving resources at training time as well. Specifically, we argue that efficient training requires preserving the gradient flow through the network. This leads to a simple but effective pruning criterion we term Gradient Signal Preservation (GraSP). We empirically investigate the effectiveness of the proposed method with extensive experiments on CIFAR-10, CIFAR-100, Tiny-ImageNet and ImageNet, using VGGNet and ResNet architectures. Our method can prune 80% of the weights of a VGG-16 network on ImageNet at initialization, with only a 1.6% drop in top-1 accuracy. Moreover, our method achieves significantly better performance than the baseline at extreme sparsity levels.

arXiv commentFix several typos

References in corpus (12)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• Rethinking the Value of Network Pruning
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science
• Sparse Networks from Scratch: Faster Training without Losing Performance
• Deep Rewiring: Training very sparse deep networks
• Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization
• Are All Layers Created Equal?
• Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks
• EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis
• The Difficulty of Training Sparse Neural Networks

Cited by in corpus (81)

• Linear Mode Connectivity and the Lottery Ticket Hypothesis
• Chasing Sparsity in Vision Transformers: An End-to-End Exploration
• Pruning neural networks without any data by iteratively conserving synaptic flow
• Zero-Cost Proxies for Lightweight NAS
• Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective
• Pruning Neural Networks at Initialization: Why are We Missing the Mark?
• Only Train Once: A One-Shot Neural Network Training And Pruning Framework
• MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edge
• Progressive Skeletonization: Trimming more fat from a network at initialization
• The Lottery Ticket Hypothesis for Pre-trained BERT Networks
• HYDRA: Pruning Adversarially Robust Neural Networks
• The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
• Evaluating Efficient Performance Estimators of Neural Architectures
• Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracy
• Rethinking Weight Decay For Efficient Neural Network Pruning
• CHIP: CHannel Independence-based Pruning for Compact Neural Networks
• A Brain-inspired Algorithm for Training Highly Sparse Neural Networks
• A Gradient Flow Framework For Analyzing Network Pruning
• Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training
• How Powerful are Performance Predictors in Neural Architecture Search?
• Network Pruning That Matters: A Case Study on Retraining Variants
• Selfish Sparse RNN Training
• Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimization
• Single Shot Structured Pruning Before Training
• Efficient Neural Network Training via Forward and Backward Propagation Sparsification
• Sparse Training via Boosting Pruning Plasticity with Neuroregeneration
• Robust Pruning at Initialization
• Pruning via Iterative Ranking of Sensitivity Statistics
• Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot
• Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity
• Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognition
• Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Network
• Bayesian Deep Learning via Subnetwork Inference
• Robustness to Pruning Predicts Generalization in Deep Neural Networks
• NASI: Label- and Data-agnostic Neural Architecture Search at Initialization
• Calibrate and Prune: Improving Reliability of Lottery Tickets Through Prediction Calibration
• Data-Efficient GAN Training Beyond (Just) Augmentations: A Lottery Ticket Perspective
• Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models
• Shapley Value as Principled Metric for Structured Network Pruning
• You are caught stealing my winning lottery ticket! Making a lottery ticket claim its ownership
• Revisiting Loss Modelling for Unstructured Pruning
• Sparse Linear Networks with a Fixed Butterfly Structure: Theory and Practice
• GANs Can Play Lottery Tickets Too
• Exploring Weight Importance and Hessian Bias in Model Pruning
• Pruning Randomly Initialized Neural Networks with Iterative Randomization
• Effective Model Sparsification by Scheduled Grow-and-Prune Methods
• The Nonlinearity Coefficient - A Practical Guide to Neural Architecture Design
• Lottery Jackpots Exist in Pre-trained Models
• Connectivity Matters: Neural Network Pruning Through the Lens of Effective Sparsity
• COPS: Controlled Pruning Before Training Starts
• Scaling Up Exact Neural Network Compression by ReLU Stability
• Dynamical Isometry: The Missing Ingredient for Neural Network Pruning
• Effective Sparsification of Neural Networks with Global Sparsity Constraint
• Finding trainable sparse networks through Neural Tangent Transfer
• Understanding the Effects of Data Parallelism and Sparsity on Neural Network Training
• Train-by-Reconnect: Decoupling Locations of Weights from their Values
• When to Prune? A Policy towards Early Structural Pruning
• FEAR: A Simple Lightweight Method to Rank Architectures
• HALP: Hardware-Aware Latency Pruning
• Initialization and Regularization of Factorized Neural Layers
• DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models
• Why Lottery Ticket Wins? A Theoretical Perspective of Sample Complexity on Pruned Neural Networks
• SOSP: Efficiently Capturing Global Correlations by Second-Order Structured Pruning
• ProxyBO: Accelerating Neural Architecture Search via Bayesian Optimization with Zero-cost Proxies
• How many degrees of freedom do we need to train deep networks: a loss landscape perspective
• Slot Machines: Discovering Winning Combinations of Random Weights in Neural Networks
• Are wider nets better given the same number of parameters?
• ESPN: Extremely Sparse Pruned Networks
• The Elastic Lottery Ticket Hypothesis
• Training highly effective connectivities within neural networks with randomly initialized, fixed weights
• Search Spaces for Neural Model Training
• BaLeNAS: Differentiable Architecture Search via the Bayesian Learning Rule
• Zero-Cost Operation Scoring in Differentiable Architecture Search
• Spending Your Winning Lottery Better After Drawing It
• Putting 3D Spatially Sparse Networks on a Diet
• Deconstructing the Structure of Sparse Neural Networks
• Probabilistic fine-tuning of pruning masks and PAC-Bayes self-bounded learning
• Connection Sensitivity Matters for Training-free DARTS: From Architecture-Level Scoring to Operation-Level Sensitivity Analysis
• Dense for the Price of Sparse: Improved Performance of Sparsely Initialized Networks via a Subspace Offset
• On the Existence of Universal Lottery Tickets
• Plant 'n' Seek: Can You Find the Winning Ticket?