CoDiNet: Path Distribution Modeling with Consistency and Diversity for Dynamic Routing
arXiv:2005.14439
Abstract
Dynamic routing networks, aimed at finding the best routing paths in the networks, have achieved significant improvements to neural networks in terms of accuracy and efficiency. In this paper, we see dynamic routing networks in a fresh light, formulating a routing method as a mapping from a sample space to a routing space. From the perspective of space mapping, prevalent methods of dynamic routing didn't consider how inference paths would be distributed in the routing space. Thus, we propose a novel method, termed CoDiNet, to model the relationship between a sample space and a routing space by regularizing the distribution of routing paths with the properties of consistency and diversity. Specifically, samples with similar semantics should be mapped into the same area in routing space, while those with dissimilar semantics should be mapped into different areas. Moreover, we design a customizable dynamic routing module, which can strike a balance between accuracy and efficiency. When deployed upon ResNet models, our method achieves higher performance and effectively reduces average computational cost on four widely used datasets.
References in corpus (12)
- Distilling the Knowledge in a Neural Network
- Categorical Reparameterization with Gumbel-Softmax
- Convolutional neural networks with low-rank regularization
- Adaptive Computation Time for Recurrent Neural Networks
- SlimYOLOv3: Narrower, Faster and Better for Real-Time UAV Applications
- Multi-Scale Dense Networks for Resource Efficient Image Classification
- Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inference
- PerforatedCNNs: Acceleration through Elimination of Redundant Convolutions
- Dual Dynamic Inference: Enabling More Efficient, Adaptive and Controllable Deep Inference
- IamNN: Iterative and Adaptive Mobile Neural Network for Efficient Image Classification
- Computation on Sparse Neural Networks: an Inspiration for Future Hardware
- TimeGate: Conditional Gating of Segments in Long-range Activities