Evolving Search Space for Neural Architecture Search

Abstract

The automation of neural architecture design has been a coveted alternative to human experts. Recent works have small search space, which is easier to optimize but has a limited upper bound of the optimal solution. Extra human design is needed for those methods to propose a more suitable space with respect to the specific task and algorithm capacity. To further enhance the degree of automation for neural architecture search, we present a Neural Search-space Evolution (NSE) scheme that iteratively amplifies the results from the previous effort by maintaining an optimized search space subset. This design minimizes the necessity of a well-designed search space. We further extend the flexibility of obtainable architectures by introducing a learnable multi-branch setting. By employing the proposed method, a consistent performance gain is achieved during a progressive search over upcoming search spaces. We achieve 77.3% top-1 retrain accuracy on ImageNet with 333M FLOPs, which yielded a state-of-the-art performance among previous auto-generated architectures that do not involve knowledge distillation or weight pruning. When the latency constraint is adopted, our result also performs better than the previous best-performing mobile models with a 77.9% Top-1 retrain accuracy.

arXiv commentAccepted for publication at the 2021 International Conference on Computer Vision (ICCV 2021)

References in corpus (20)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Neural Architecture Search with Reinforcement Learning
• Searching for Activation Functions
• Efficient Neural Architecture Search via Parameter Sharing
• Large-Scale Evolution of Image Classifiers
• MixConv: Mixed Depthwise Convolutional Kernels
• Slimmable Neural Networks
• A Survey on Neural Architecture Search
• NAS evaluation is frustratingly hard
• FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search
• What is the State of Neural Network Pruning?
• NSGA-Net: Neural Architecture Search using Multi-Objective Genetic Algorithm
• You Only Search Once: Single Shot Neural Architecture Search via Direct Sparse Optimization
• Computation Reallocation for Object Detection
• FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining
• Angle-based Search Space Shrinking for Neural Architecture Search
• BETANAS: BalancEd TrAining and selective drop for Neural Architecture Search
• Fast Hardware-Aware Neural Architecture Search
• How Does Supernet Help in Neural Architecture Search?
• FP-NAS: Fast Probabilistic Neural Architecture Search

Cited by in corpus (2)

• PSViT: Better Vision Transformer via Token Pooling and Attention Sharing
• Integrating Large Circular Kernels into CNNs through Neural Architecture Search