Progressive Skeletonization: Trimming more fat from a network at initialization

Abstract

Recent studies have shown that skeletonization (pruning parameters) of networks \textit{at initialization} provides all the practical benefits of sparsity both at inference and training time, while only marginally degrading their performance. However, we observe that beyond a certain level of sparsity (approx $95\%$), these approaches fail to preserve the network performance, and to our surprise, in many cases perform even worse than trivial random pruning. To this end, we propose an objective to find a skeletonized network with maximum {\em foresight connection sensitivity} (FORCE) whereby the trainability, in terms of connection sensitivity, of a pruned network is taken into consideration. We then propose two approximate procedures to maximize our objective (1) Iterative SNIP: allows parameters that were unimportant at earlier stages of skeletonization to become important at later stages; and (2) FORCE: iterative process that allows exploration by allowing already pruned parameters to resurrect at later stages of skeletonization. Empirical analyses on a large suite of experiments show that our approach, while providing at least as good a performance as other recent approaches on moderate pruning levels, provides remarkably improved performance on higher pruning levels (could remove up to $99.5\%$ parameters while keeping the networks trainable). Code can be found in https://github.com/naver/force.

References in corpus (14)

• Neural Architecture Search with Reinforcement Learning
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• Rethinking the Value of Network Pruning
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science
• Sparse Networks from Scratch: Faster Training without Losing Performance
• Deep Rewiring: Training very sparse deep networks
• Picking Winning Tickets Before Training by Preserving Gradient Flow
• Stabilizing the Lottery Ticket Hypothesis
• Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization
• Rigging the Lottery: Making All Tickets Winners
• Soft Threshold Weight Reparameterization for Learnable Sparsity
• Dynamic Model Pruning with Feedback
• Pruning via Iterative Ranking of Sensitivity Statistics

Cited by in corpus (8)

• Pruning neural networks without any data by iteratively conserving synaptic flow
• Pruning Neural Networks at Initialization: Why are We Missing the Mark?
• A Brain-inspired Algorithm for Training Highly Sparse Neural Networks
• Selfish Sparse RNN Training
• When to Prune? A Policy towards Early Structural Pruning
• SEVEN: Pruning Transformer Model by Reserving Sentinels
• HALP: Hardware-Aware Latency Pruning
• Optimizing Connectivity through Network Gradients for Restricted Boltzmann Machines