Distilling with Performance Enhanced Students
arXiv:1810.10460
Abstract
The task of accelerating large neural networks on general purpose hardware has, in recent years, prompted the use of channel pruning to reduce network size. However, the efficacy of pruning based approaches has since been called into question. In this paper, we turn to distillation for model compression---specifically, attention transfer---and develop a simple method for discovering performance enhanced student networks. We combine channel saliency metrics with empirical observations of runtime performance to design more accurate networks for a given latency budget. We apply our methodology to residual and densely-connected networks, and show that we are able to find resource-efficient student networks on different hardware platforms while maintaining very high accuracy. These performance-enhanced student networks achieve up to 10% boosts in top-1 ImageNet accuracy over their channel-pruned counterparts for the same inference time.
Preprint. Paper title has changed
References in corpus (13)
- Distilling the Knowledge in a Neural Network
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
- Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
- Do Deep Nets Really Need to be Deep?
- Pruning Convolutional Neural Networks for Resource Efficient Inference
- AMC: AutoML for Model Compression and Acceleration on Mobile Devices
- cuDNN: Efficient Primitives for Deep Learning
- Rethinking the Value of Network Pruning
- Bayesian Compression for Deep Learning
- Faster gaze prediction with dense networks and Fisher pruning
- Moonshine: Distilling with Cheap Convolutions
- DPP-Net: Device-aware Progressive Search for Pareto-optimal Neural Architectures