A Signal Propagation Perspective for Pruning Neural Networks at Initialization

Abstract

Network pruning is a promising avenue for compressing deep neural networks. A typical approach to pruning starts by training a model and then removing redundant parameters while minimizing the impact on what is learned. Alternatively, a recent approach shows that pruning can be done at initialization prior to training, based on a saliency criterion called connection sensitivity. However, it remains unclear exactly why pruning an untrained, randomly initialized neural network is effective. In this work, by noting connection sensitivity as a form of gradient, we formally characterize initialization conditions to ensure reliable connection sensitivity measurements, which in turn yields effective pruning results. Moreover, we analyze the signal propagation properties of the resulting pruned networks and introduce a simple, data-free method to improve their trainability. Our modifications to the existing pruning at initialization method lead to improved results on all tested network models for image classification tasks. Furthermore, we empirically study the effect of supervision for pruning and demonstrate that our signal propagation perspective, combined with unsupervised pruning, can be useful in various scenarios where pruning is applied to non-standard arbitrarily-designed architectures.

arXiv commentICLR 2020

References in corpus (4)

• On the difficulty of training Recurrent Neural Networks
• Exponential expressivity in deep neural networks through transient chaos
• The Emergence of Spectral Universality in Deep Networks
• Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks

Cited by in corpus (6)

• Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimization
• F3-Pruning: A Training-Free and Generalized Pruning Strategy towards Faster and Finer Text-to-Video Synthesis
• Revisiting Loss Modelling for Unstructured Pruning
• Know What You Don't Need: Single-Shot Meta-Pruning for Attention Heads
• The Heterogeneity Hypothesis: Finding Layer-Wise Differentiated Network Architectures
• Statistical Mechanical Analysis of Neural Network Pruning