Densely Connected Search Space for More Flexible Neural Architecture Search

Abstract

Neural architecture search (NAS) has dramatically advanced the development of neural network design. We revisit the search space design in most previous NAS methods and find the number and widths of blocks are set manually. However, block counts and block widths determine the network scale (depth and width) and make a great influence on both the accuracy and the model cost (FLOPs/latency). In this paper, we propose to search block counts and block widths by designing a densely connected search space, i.e., DenseNAS. The new search space is represented as a dense super network, which is built upon our designed routing blocks. In the super network, routing blocks are densely connected and we search for the best path between them to derive the final architecture. We further propose a chained cost estimation algorithm to approximate the model cost during the search. Both the accuracy and model cost are optimized in DenseNAS. For experiments on the MobileNetV2-based search space, DenseNAS achieves 75.3% top-1 accuracy on ImageNet with only 361MB FLOPs and 17.9ms latency on a single TITAN-XP. The larger model searched by DenseNAS achieves 76.1% accuracy with only 479M FLOPs. DenseNAS further promotes the ImageNet classification accuracies of ResNet-18, -34 and -50-B by 1.5%, 0.5% and 0.3% with 200M, 600M and 680M FLOPs reduction respectively. The related code is available at https://github.com/JaminFong/DenseNAS.

arXiv commentAccepted by CVPR 2020

References in corpus (17)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Neural Architecture Search with Reinforcement Learning
• Categorical Reparameterization with Gumbel-Softmax
• DARTS: Differentiable Architecture Search
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• MMDetection: Open MMLab Detection Toolbox and Benchmark
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• Searching for Efficient Multi-Scale Architectures for Dense Image Prediction
• SNAS: Stochastic Neural Architecture Search
• Random Search and Reproducibility for Neural Architecture Search
• Network Pruning via Transformable Architecture Search
• FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search
• VarGNet: Variable Group Convolutional Neural Network for Efficient Embedded Computing
• ChamNet: Towards Efficient Network Design through Platform-Aware Model Adaptation
• Fast Neural Network Adaptation via Parameter Remapping and Architecture Search

Cited by in corpus (7)

• Hierarchical Neural Architecture Search for Deep Stereo Matching
• One-Shot Neural Architecture Search via Self-Evaluated Template Network
• Bringing AI To Edge: From Deep Learning's Perspective
• Towards Improving the Consistency, Efficiency, and Flexibility of Differentiable Neural Architecture Search
• Effective and Fast: A Novel Sequential Single Path Search for Mixed-Precision Quantization
• Fitting the Search Space of Weight-sharing NAS with Graph Convolutional Networks
• 3D Matting: A Benchmark Study on Soft Segmentation Method for Pulmonary Nodules Applied in Computed Tomography