BlockSwap: Fisher-guided Block Substitution for Network Compression on a Budget
arXiv:1906.04113
Abstract
The desire to map neural networks to varying-capacity devices has led to the development of a wealth of compression techniques, many of which involve replacing standard convolutional blocks in a large network with cheap alternative blocks. However, not all blocks are created equally; for a required compute budget there may exist a potent combination of many different cheap blocks, though exhaustively searching for such a combination is prohibitively expensive. In this work, we develop BlockSwap: a fast algorithm for choosing networks with interleaved block types by passing a single minibatch of training data through randomly initialised networks and gauging their Fisher potential. These networks can then be used as students and distilled with the original large network as a teacher. We demonstrate the effectiveness of the chosen networks across CIFAR-10 and ImageNet for classification, and COCO for detection, and provide a comprehensive ablation study of our approach. BlockSwap quickly explores possible block configurations using a simple architecture ranking system, yielding highly competitive networks in orders of magnitude less time than most architecture search techniques (e.g. under 5 minutes on a single GPU for CIFAR-10). Code is available at https://github.com/BayesWatch/pytorch-blockswap.
arXiv commentICLR 2020
References in corpus (20)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Distilling the Knowledge in a Neural Network
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Neural Architecture Search with Reinforcement Learning
- Improved Regularization of Convolutional Neural Networks with Cutout
- Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
- Do Deep Nets Really Need to be Deep?
- DARTS: Differentiable Architecture Search
- Pruning Convolutional Neural Networks for Resource Efficient Inference
- Rethinking the Value of Network Pruning
- SNIP: Single-shot Network Pruning based on Connection Sensitivity
- Neural Architecture Optimization
- Searching for Efficient Multi-Scale Architectures for Dense Image Prediction
- Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups
- Random Search and Reproducibility for Neural Architecture Search
- Loss-aware Binarization of Deep Networks
- Evaluating the Search Phase of Neural Architecture Search
- Faster gaze prediction with dense networks and Fisher pruning
- Moonshine: Distilling with Cheap Convolutions
- Accelerating Deep Neural Networks with Spatial Bottleneck Modules