Hierarchical Neural Architecture Search via Operator Clustering

Abstract

Recently, the efficiency of automatic neural architecture design has been significantly improved by gradient-based search methods such as DARTS. However, recent literature has brought doubt to the generalization ability of DARTS, arguing that DARTS performs poorly when the search space is changed, i.e, when different set of candidate operators are used. Regularization techniques such as early stopping have been proposed to partially solve this problem. In this paper, we tackle this problem from a different perspective by identifying two contributing factors to the collapse of DARTS when the search space changes: (1) the correlation of similar operators incurs unfavorable competition among them and makes their relative importance score unreliable and (2) the optimization complexity gap between the proxy search stage and the final training. Based on these findings, we propose a new hierarchical search algorithm. With its operator clustering and optimization complexity match, the algorithm can consistently find high-performance architecture across various search spaces. For all the five variants of the popular cell-based search spaces, the proposed algorithm always obtains state-of-the-art architecture with best accuracy on the CIFAR-10, CIFAR-100 and ImageNet over other well-established DARTS-alike algorithms. Code is available at https://github.com/susan0199/StacNAS.

References in corpus (15)

• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Neural Architecture Search with Reinforcement Learning
• DARTS: Differentiable Architecture Search
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• AutoAugment: Learning Augmentation Policies from Data
• Efficient Neural Architecture Search via Parameter Sharing
• PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
• Shake-Shake regularization
• Understanding and Robustifying Differentiable Architecture Search
• DARTS+: Improved Differentiable Architecture Search with Early Stopping
• Random Search and Reproducibility for Neural Architecture Search
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• AutoFIS: Automatic Feature Interaction Selection in Factorization Models for Click-Through Rate Prediction
• The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent
• XNAS: Neural Architecture Search with Expert Advice

Cited by in corpus (7)

• DARTS+: Improved Differentiable Architecture Search with Early Stopping
• ResKD: Residual-Guided Knowledge Distillation
• AutoFIS: Automatic Feature Interaction Selection in Factorization Models for Click-Through Rate Prediction
• GOLD-NAS: Gradual, One-Level, Differentiable
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• DOTS: Decoupling Operation and Topology in Differentiable Architecture Search
• Single-level Optimization For Differential Architecture Search