Pruning Neural Networks at Initialization: Why are We Missing the Mark?

Abstract

Recent work has explored the possibility of pruning neural networks at initialization. We assess proposals for doing so: SNIP (Lee et al., 2019), GraSP (Wang et al., 2020), SynFlow (Tanaka et al., 2020), and magnitude pruning. Although these methods surpass the trivial baseline of random pruning, they remain below the accuracy of magnitude pruning after training, and we endeavor to understand why. We show that, unlike pruning after training, randomly shuffling the weights these methods prune within each layer or sampling new initial values preserves or improves accuracy. As such, the per-weight pruning decisions made by these methods can be replaced by a per-layer choice of the fraction of weights to prune. This property suggests broader challenges with the underlying pruning heuristics, the desire to prune at initialization, or both.

arXiv commentPublished in ICLR 2021

References in corpus (18)

• Language Models are Few-Shot Learners
• Wide Residual Networks
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science
• The State of Sparsity in Deep Neural Networks
• Sparse Networks from Scratch: Faster Training without Losing Performance
• Comparing Rewinding and Fine-tuning in Neural Network Pruning
• Picking Winning Tickets Before Training by Preserving Gradient Flow
• Gradient Descent Happens in a Tiny Subspace
• Norm-Based Capacity Control in Neural Networks
• The Early Phase of Neural Network Training
• Progressive Skeletonization: Trimming more fat from a network at initialization
• The Break-Even Point on Optimization Trajectories of Deep Neural Networks
• The Difficulty of Training Sparse Neural Networks
• Pruning via Iterative Ranking of Sensitivity Statistics
• ESPN: Extremely Sparse Pruned Networks

Cited by in corpus (8)

• Real-time Neural Network Inference on Extremely Weak Devices: Agile Offloading with Explainable AI
• Complexity-Driven CNN Compression for Resource-constrained Edge AI
• Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimization
• OnDev-LCT: On-Device Lightweight Convolutional Transformers towards federated learning
• Deep learning via message passing algorithms based on belief propagation
• Pruning a restricted Boltzmann machine for quantum state reconstruction
• Are wider nets better given the same number of parameters?
• Neural Networks at a Fraction with Pruned Quaternions