Pruning neural networks without any data by iteratively conserving synaptic flow

Abstract

Pruning the parameters of deep neural networks has generated intense interest due to potential savings in time, memory and energy both during training and at test time. Recent works have identified, through an expensive sequence of training and pruning cycles, the existence of winning lottery tickets or sparse trainable subnetworks at initialization. This raises a foundational question: can we identify highly sparse trainable subnetworks at initialization, without ever training, or indeed without ever looking at the data? We provide an affirmative answer to this question through theory driven algorithm design. We first mathematically formulate and experimentally verify a conservation law that explains why existing gradient-based pruning algorithms at initialization suffer from layer-collapse, the premature pruning of an entire layer rendering a network untrainable. This theory also elucidates how layer-collapse can be entirely avoided, motivating a novel pruning algorithm Iterative Synaptic Flow Pruning (SynFlow). This algorithm can be interpreted as preserving the total flow of synaptic strengths through the network at initialization subject to a sparsity constraint. Notably, this algorithm makes no reference to the training data and consistently competes with or outperforms existing state-of-the-art pruning algorithms at initialization over a range of models (VGG and ResNet), datasets (CIFAR-10/100 and Tiny ImageNet), and sparsity constraints (up to 99.99 percent). Thus our data-agnostic pruning algorithm challenges the existing paradigm that, at initialization, data must be used to quantify which synapses are important.

NeurIPS 2020, 18 pages, 10 figures

References in corpus (24)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• Rethinking the Value of Network Pruning
• Tensorizing Neural Networks
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science
• The State of Sparsity in Deep Neural Networks
• Pruning by Explaining: A Novel Criterion for Deep Neural Network Pruning
• L2 Regularization versus Batch and Weight Normalization
• Path-SGD: Path-Normalized Optimization in Deep Neural Networks
• Deep Rewiring: Training very sparse deep networks
• Picking Winning Tickets Before Training by Preserving Gradient Flow
• Stabilizing the Lottery Ticket Hypothesis
• Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization
• Rigging the Lottery: Making All Tickets Winners
• Fisher-Rao Metric, Geometry, and Complexity of Neural Networks
• Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP
• What is the State of Neural Network Pruning?
• How Important Is a Neuron?
• From deep learning to mechanistic understanding in neuroscience: the structure of retinal prediction
• Progressive Skeletonization: Trimming more fat from a network at initialization
• Lookahead: A Far-Sighted Alternative of Magnitude-based Pruning
• Pruning via Iterative Ranking of Sensitivity Statistics

Cited by in corpus (20)

• Neural Architecture Search without Training
• Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracy
• A Brain-inspired Algorithm for Training Highly Sparse Neural Networks
• NeurDB: An AI-powered Autonomous Data System
• A Gradient Flow Framework For Analyzing Network Pruning
• Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training
• Selfish Sparse RNN Training
• Dynamic Sparse Learning: A Novel Paradigm for Efficient Recommendation
• Learning Neural Network Subspaces
• A Unified Paths Perspective for Pruning at Initialization
• Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics
• GradSign: Model Performance Inference with Theoretical Insights
• Synaptic balancing: a biologically plausible local learning rule that provably increases neural network noise robustness without sacrificing task performance
• When to Prune? A Policy towards Early Structural Pruning
• AIO-P: Expanding Neural Performance Predictors Beyond Image Classification
• The Elastic Lottery Ticket Hypothesis
• Why is Pruning at Initialization Immune to Reinitializing and Shuffling?
• A brain basis of dynamical intelligence for AI and computational neuroscience
• Deconstructing the Structure of Sparse Neural Networks
• Probabilistic fine-tuning of pruning masks and PAC-Bayes self-bounded learning