Dynamic Routing Networks
arXiv:1905.04849
Abstract
The deployment of deep neural networks in real-world applications is mostly restricted by their high inference costs. Extensive efforts have been made to improve the accuracy with expert-designed or algorithm-searched architectures. However, the incremental improvement is typically achieved with increasingly more expensive models that only a small portion of input instances really need. Inference with a static architecture that processes all input instances via the same transformation would thus incur unnecessary computational costs. Therefore, customizing the model capacity in an instance-aware manner is much needed for higher inference efficiency. In this paper, we propose Dynamic Routing Networks (DRNets), which support efficient instance-aware inference by routing the input instance to only necessary transformation branches selected from a candidate set of branches for each connection between transformation nodes. The branch selection is dynamically determined via the corresponding branch importance weights, which are first generated from lightweight hypernetworks (RouterNets) and then recalibrated with Gumbel-Softmax before the selection. Extensive experiments show that DRNets can reduce a substantial amount of parameter size and FLOPs during inference with prediction performance comparable to state-of-the-art architectures.
arXiv comment10 pages, 3 figures, 3 tables
References in corpus (14)
- Auto-Encoding Variational Bayes
- Distilling the Knowledge in a Neural Network
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
- Neural Architecture Search with Reinforcement Learning
- Categorical Reparameterization with Gumbel-Softmax
- Improved Regularization of Convolutional Neural Networks with Cutout
- DARTS: Differentiable Architecture Search
- The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
- AutoAugment: Learning Augmentation Policies from Data
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
- Multi-Scale Dense Networks for Resource Efficient Image Classification
- Exploring Randomly Wired Neural Networks for Image Recognition
- Modular Networks: Learning to Decompose Neural Computation