SNIP: Single-shot Network Pruning based on Connection Sensitivity

Abstract

Pruning large neural networks while maintaining their performance is often desirable due to the reduced space and time complexity. In existing methods, pruning is done within an iterative optimization procedure with either heuristically designed pruning schedules or additional hyperparameters, undermining their utility. In this work, we present a new approach that prunes a given network once at initialization prior to training. To achieve this, we introduce a saliency criterion based on connection sensitivity that identifies structurally important connections in the network for the given task. This eliminates the need for both pretraining and the complex pruning schedule while making it robust to architecture variations. After pruning, the sparse network is trained in the standard way. Our method obtains extremely sparse networks with virtually the same accuracy as the reference network on the MNIST, CIFAR-10, and Tiny-ImageNet classification tasks and is broadly applicable to various architectures including convolutional, residual and recurrent networks. Unlike existing methods, our approach enables us to demonstrate that the retained connections are indeed relevant to the given task.

arXiv commentICLR 2019

References in corpus (9)

• Recurrent Neural Network Regularization
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• Compressing Deep Convolutional Networks using Vector Quantization
• Pruning Filters for Efficient ConvNets
• Tensorizing Neural Networks
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Learning Structured Sparsity in Deep Neural Networks
• Dynamic Network Surgery for Efficient DNNs
• Exploring Sparsity in Recurrent Neural Networks

Cited by in corpus (145)

• Machine Learning for Microcontroller-Class Hardware: A Review
• Sparse Networks from Scratch: Faster Training without Losing Performance
• Adaptive Extreme Edge Computing for Wearable Devices
• Discrimination-aware Network Pruning for Deep Model Compression
• AutoSlim: Towards One-Shot Architecture Search for Channel Numbers
• Picking Winning Tickets Before Training by Preserving Gradient Flow
• Stabilizing the Lottery Ticket Hypothesis
• Rigging the Lottery: Making All Tickets Winners
• Soft Threshold Weight Reparameterization for Learnable Sparsity
• Linear Mode Connectivity and the Lottery Ticket Hypothesis
• What Do Compressed Deep Neural Networks Forget?
• Applications and Techniques for Fast Machine Learning in Science
• A Low Effort Approach to Structured CNN Design Using PCA
• Pruning neural networks without any data by iteratively conserving synaptic flow
• Layer-adaptive sparsity for the Magnitude-based Pruning
• FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout
• Optimizing Speech Recognition For The Edge
• Compacting Deep Neural Networks for Internet of Things: Methods and Applications
• The learnability scaling of quantum states: restricted Boltzmann machines
• Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks
• Pruning Neural Networks at Initialization: Why are We Missing the Mark?
• Only Train Once: A One-Shot Neural Network Training And Pruning Framework
• Model Pruning Enables Localized and Efficient Federated Learning for Yield Forecasting and Data Sharing
• ExpandNets: Linear Over-parameterization to Train Compact Convolutional Networks
• Progressive Skeletonization: Trimming more fat from a network at initialization
• Complexity-Driven CNN Compression for Resource-constrained Edge AI
• When BERT Plays the Lottery, All Tickets Are Winning
• The Lottery Ticket Hypothesis for Pre-trained BERT Networks
• Movement Pruning: Adaptive Sparsity by Fine-Tuning
• Gradual Channel Pruning while Training using Feature Relevance Scores for Convolutional Neural Networks
• HYDRA: Pruning Adversarially Robust Neural Networks
• Evaluating Efficient Performance Estimators of Neural Architectures
• LOss-Based SensiTivity rEgulaRization: towards deep sparse neural networks
• FreeREA: Training-Free Evolution-based Architecture Search
• Rethinking Weight Decay For Efficient Neural Network Pruning
• CHIP: CHannel Independence-based Pruning for Compact Neural Networks
• Provable Filter Pruning for Efficient Neural Networks
• End-to-End Supermask Pruning: Learning to Prune Image Captioning Models
• Model Pruning Based on Quantified Similarity of Feature Maps
• Loss-Aware Automatic Selection of Structured Pruning Criteria for Deep Neural Network Acceleration
• Accelerate CNNs from Three Dimensions: A Comprehensive Pruning Framework
• A Closer Look at Structured Pruning for Neural Network Compression
• A Brain-inspired Algorithm for Training Highly Sparse Neural Networks
• Perceptron Theory Can Predict the Accuracy of Neural Networks
• A Gradient Flow Framework For Analyzing Network Pruning
• Sparse Transfer Learning via Winning Lottery Tickets
• SeReNe: Sensitivity based Regularization of Neurons for Structured Sparsity in Neural Networks
• Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training
• Network Pruning That Matters: A Case Study on Retraining Variants
• How Powerful are Performance Predictors in Neural Architecture Search?
• Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimization
• Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?
• FreezeNet: Full Performance by Reduced Storage Costs
• Modeling of Pruning Techniques for Deep Neural Networks Simplification
• SiPPing Neural Networks: Sensitivity-informed Provable Pruning of Neural Networks
• Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks
• One-Shot Pruning of Recurrent Neural Networks by Jacobian Spectrum Evaluation
• Observation Space Matters: Benchmark and Optimization Algorithm
• DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures
• Neural network relief: a pruning algorithm based on neural activity
• BlockSwap: Fisher-guided Block Substitution for Network Compression on a Budget
• Robust Pruning at Initialization
• Sparse Training via Boosting Pruning Plasticity with Neuroregeneration
• Real-time Universal Style Transfer on High-resolution Images via Zero-channel Pruning
• Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot
• Neuron Merging: Compensating for Pruned Neurons
• Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity
• AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networks
• Logic Shrinkage: Learned FPGA Netlist Sparsity for Efficient Neural Network Inference
• Mirror Descent View for Neural Network Quantization
• Towards Practical Lottery Ticket Hypothesis for Adversarial Training
• NASI: Label- and Data-agnostic Neural Architecture Search at Initialization
• Composition of Saliency Metrics for Channel Pruning with a Myopic Oracle
• Exploring Gradient Flow Based Saliency for DNN Model Compression
• Powerpropagation: A sparsity inducing weight reparameterisation
• PCNN: Pattern-based Fine-Grained Regular Pruning towards Optimizing CNN Accelerators
• On improving deep learning generalization with adaptive sparse connectivity
• Intrinsically Sparse Long Short-Term Memory Networks
• You are caught stealing my winning lottery ticket! Making a lottery ticket claim its ownership
• GradSign: Model Performance Inference with Theoretical Insights
• Effective Model Sparsification by Scheduled Grow-and-Prune Methods
• Sparse Linear Networks with a Fixed Butterfly Structure: Theory and Practice
• Hessian-Aware Pruning and Optimal Neural Implant
• Pruning Randomly Initialized Neural Networks with Iterative Randomization
• A General-Purpose Transferable Predictor for Neural Architecture Search
• Effective Sparsification of Neural Networks with Global Sparsity Constraint
• Dynamical Isometry: The Missing Ingredient for Neural Network Pruning
• Dynamic Group Convolution for Accelerating Convolutional Neural Networks
• Topological Insights into Sparse Neural Networks
• Lottery Jackpots Exist in Pre-trained Models
• Single-shot Channel Pruning Based on Alternating Direction Method of Multipliers
• COPS: Controlled Pruning Before Training Starts
• Scaling Up Exact Neural Network Compression by ReLU Stability
• Connectivity Matters: Neural Network Pruning Through the Lens of Effective Sparsity
• Identifying Critical Neurons in ANN Architectures using Mixed Integer Programming
• Train-by-Reconnect: Decoupling Locations of Weights from their Values
• Understanding the Effects of Data Parallelism and Sparsity on Neural Network Training
• LNPT: Label-free Network Pruning and Training
• A Tunable Robust Pruning Framework Through Dynamic Network Rewiring of DNNs
• Image Captioning with Sparse Recurrent Neural Network
• Activation Density driven Energy-Efficient Pruning in Training
• Sparse Uncertainty Representation in Deep Learning with Inducing Weights
• ProxyBO: Accelerating Neural Architecture Search via Bayesian Optimization with Zero-cost Proxies
• Are wider nets better given the same number of parameters?
• Adaptive Pixel-wise Structured Sparse Network for Efficient CNNs
• Initialization and Regularization of Factorized Neural Layers
• Dynamic Sparsity Neural Networks for Automatic Speech Recognition
• How many degrees of freedom do we need to train deep networks: a loss landscape perspective
• Slot Machines: Discovering Winning Combinations of Random Weights in Neural Networks
• SOSP: Efficiently Capturing Global Correlations by Second-Order Structured Pruning
• DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models
• PrunedCaps: A Case For Primary Capsules Discrimination
• Stochastic Model Pruning via Weight Dropping Away and Back
• Spectral Pruning for Recurrent Neural Networks
• Why is Pruning at Initialization Immune to Reinitializing and Shuffling?
• BaLeNAS: Differentiable Architecture Search via the Bayesian Learning Rule
• Using noise to probe recurrent neural network structure and prune synapses
• The Elastic Lottery Ticket Hypothesis
• Graph Pruning for Model Compression
• EPIC TTS Models: Empirical Pruning Investigations Characterizing Text-To-Speech Models
• Class-dependent Compression of Deep Neural Networks
• Yield Loss Reduction and Test of AI and Deep Learning Accelerators
• Training highly effective connectivities within neural networks with randomly initialized, fixed weights
• Zero-Cost Operation Scoring in Differentiable Architecture Search
• The Heterogeneity Hypothesis: Finding Layer-Wise Differentiated Network Architectures
• Neural Parameter Allocation Search
• Does a sparse ReLU network training problem always admit an optimum?
• Extremely Low Footprint End-to-End ASR System for Smart Device
• Effective Model Compression via Stage-wise Pruning
• Coarse and fine-grained automatic cropping deep convolutional neural network
• Induced Feature Selection by Structured Pruning
• Distilling Image Classifiers in Object Detectors
• Statistical Mechanical Analysis of Neural Network Pruning
• Connection Sensitivity Matters for Training-free DARTS: From Architecture-Level Scoring to Operation-Level Sensitivity Analysis
• One-Cycle Pruning: Pruning ConvNets Under a Tight Training Budget
• HALO: Learning to Prune Neural Networks with Shrinkage
• Structured Ensembles: an Approach to Reduce the Memory Footprint of Ensemble Methods
• Dense for the Price of Sparse: Improved Performance of Sparsely Initialized Networks via a Subspace Offset
• Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers
• On the Existence of Universal Lottery Tickets
• Plant 'n' Seek: Can You Find the Winning Ticket?
• A Survey on Green Deep Learning
• RGP: Neural Network Pruning through Its Regular Graph Structure
• RNN Architecture Learning with Sparse Regularization
• Optimizing Connectivity through Network Gradients for Restricted Boltzmann Machines