Robust Pruning at Initialization
arXiv:2002.08797
Abstract
Overparameterized Neural Networks (NN) display state-of-the-art performance. However, there is a growing need for smaller, energy-efficient, neural networks tobe able to use machine learning applications on devices with limited computational resources. A popular approach consists of using pruning techniques. While these techniques have traditionally focused on pruning pre-trained NN (LeCun et al.,1990; Hassibi et al., 1993), recent work by Lee et al. (2018) has shown promising results when pruning at initialization. However, for Deep NNs, such procedures remain unsatisfactory as the resulting pruned networks can be difficult to train and, for instance, they do not prevent one layer from being fully pruned. In this paper, we provide a comprehensive theoretical analysis of Magnitude and Gradient based pruning at initialization and training of sparse architectures. This allows us to propose novel principled approaches which we validate experimentally on a variety of NN architectures.
arXiv comment37 pages, 12 figures
References in corpus (11)
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- SNIP: Single-shot Network Pruning based on Connection Sensitivity
- Deep Neural Networks as Gaussian Processes
- Gradient Descent Provably Optimizes Over-parameterized Neural Networks
- Exponential expressivity in deep neural networks through transient chaos
- Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
- Picking Winning Tickets Before Training by Preserving Gradient Flow
- Compression-aware Training of Deep Networks
- On the Impact of the Activation Function on Deep Neural Networks Training
- A Mean Field Theory of Batch Normalization
- Optimization Landscape and Expressivity of Deep CNNs