Rethinking Bottleneck Structure for Efficient Mobile Network Design
arXiv:2007.02269
Abstract
The inverted residual block is dominating architecture design for mobile networks recently. It changes the classic residual bottleneck by introducing two design rules: learning inverted residuals and using linear bottlenecks. In this paper, we rethink the necessity of such design changes and find it may bring risks of information loss and gradient confusion. We thus propose to flip the structure and present a novel bottleneck design, called the sandglass block, that performs identity mapping and spatial transformation at higher dimensions and thus alleviates information loss and gradient confusion effectively. Extensive experiments demonstrate that, different from the common belief, such bottleneck structure is more beneficial than the inverted ones for mobile networks. In ImageNet classification, by simply replacing the inverted residual block with our sandglass block without increasing parameters and computation, the classification accuracy can be improved by more than 1.7% over MobileNetV2. On Pascal VOC 2007 test set, we observe that there is also 0.9% mAP improvement in object detection. We further verify the effectiveness of the sandglass block by adding it into the search space of neural architecture search method DARTS. With 25% parameter reduction, the classification accuracy is improved by 0.13% over previous DARTS models. Code can be found at: https://github.com/zhoudaquan/rethinking_bottleneck_design.
arXiv commentA journal version under review of the previous paper published as a ECCV20 conference paper, improved segmentation and detection results
References in corpus (16)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Distilling the Knowledge in a Neural Network
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Wide Residual Networks
- Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
- Speeding up Convolutional Neural Networks with Low Rank Expansions
- MixConv: Mixed Depthwise Convolutional Kernels
- NAS-Bench-101: Towards Reproducible Neural Architecture Search
- Single Path One-Shot Neural Architecture Search with Uniform Sampling
- GhostNet: More Features from Cheap Operations
- NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search
- Dual Path Networks
- The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent
- Pruning Convolutional Neural Networks with Self-Supervision
- IGCV3: Interleaved Low-Rank Group Convolutions for Efficient Deep Neural Networks