On Iterative Neural Network Pruning, Reinitialization, and the Similarity of Masks
arXiv:2001.05050
Abstract
We examine how recently documented, fundamental phenomena in deep learning models subject to pruning are affected by changes in the pruning procedure. Specifically, we analyze differences in the connectivity structure and learning dynamics of pruned models found through a set of common iterative pruning techniques, to address questions of uniqueness of trainable, high-sparsity sub-networks, and their dependence on the chosen pruning method. In convolutional layers, we document the emergence of structure induced by magnitude-based unstructured pruning in conjunction with weight rewinding that resembles the effects of structured pruning. We also show empirical evidence that weight stability can be automatically achieved through apposite pruning techniques.
8 pages, 8 figures, plus 5 appendices with additional figures and tables
References in corpus (6)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- The State of Sparsity in Deep Neural Networks
- Dynamic Network Surgery for Efficient DNNs
- ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression
- Exploring Randomly Wired Neural Networks for Image Recognition
- Fine-Pruning: Joint Fine-Tuning and Compression of a Convolutional Network with Bayesian Optimization