Activation Density driven Energy-Efficient Pruning in Training
arXiv:2002.02949
Abstract
Neural network pruning with suitable retraining can yield networks with considerably fewer parameters than the original with comparable degrees of accuracy. Typical pruning methods require large, fully trained networks as a starting point from which they perform a time-intensive iterative pruning and retraining procedure to regain the original accuracy. We propose a novel pruning method that prunes a network real-time during training, reducing the overall training time to achieve an efficient compressed network. We introduce an activation density based analysis to identify the optimal relative sizing or compression for each layer of the network. Our method is architecture agnostic, allowing it to be employed on a wide variety of systems. For VGG-19 and ResNet18 on CIFAR-10, CIFAR-100, and TinyImageNet, we obtain exceedingly sparse networks (up to reduction in parameters and over reduction in inference compute operations in the best case) with accuracy comparable to the baseline network. By reducing the network size periodically during training, we achieve total training times that are shorter than those of previously proposed pruning methods. Furthermore, training compressed networks at different epochs with our proposed method yields considerable reduction in training compute complexity ( to lower) at near iso-accuracy as compared to a baseline network trained entirely from scratch.
8 pages, 5 figures, 4 tables (Accepted in ICPR 2020)
References in corpus (8)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
- Rethinking the Value of Network Pruning
- SNIP: Single-shot Network Pruning based on Connection Sensitivity
- Learning Structured Sparsity in Deep Neural Networks
- Stabilizing the Lottery Ticket Hypothesis
- A Low Effort Approach to Structured CNN Design Using PCA
- Dynamic Sparse Graph for Efficient Deep Learning