Angle-based Search Space Shrinking for Neural Architecture Search

Abstract

In this work, we present a simple and general search space shrinking method, called Angle-Based search space Shrinking (ABS), for Neural Architecture Search (NAS). Our approach progressively simplifies the original search space by dropping unpromising candidates, thus can reduce difficulties for existing NAS methods to find superior architectures. In particular, we propose an angle-based metric to guide the shrinking process. We provide comprehensive evidences showing that, in weight-sharing supernet, the proposed metric is more stable and accurate than accuracy-based and magnitude-based metrics to predict the capability of child models. We also show that the angle-based metric can converge fast while training supernet, enabling us to get promising shrunk search spaces efficiently. ABS can easily apply to most of NAS approaches (e.g. SPOS, FairNAS, ProxylessNAS, DARTS and PDARTS). Comprehensive experiments show that ABS can dramatically enhance existing NAS approaches by providing a promising shrunk search space.

Accepted in ECCV 2020

References in corpus (12)

• Searching for Activation Functions
• Efficient Neural Architecture Search via Parameter Sharing
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• One-Shot Neural Architecture Search via Self-Evaluated Template Network
• Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation
• Theoretical Analysis of Auto Rate-Tuning by Batch Normalization
• ASAP: Architecture Search, Anneal and Prune
• An Exponential Learning Rate Schedule for Deep Learning
• Deeper Insights into Weight Sharing in Neural Architecture Search
• Understanding Neural Architecture Search Techniques
• Improving One-shot NAS by Suppressing the Posterior Fading
• Spherical Motion Dynamics: Learning Dynamics of Neural Network with Normalization, Weight Decay, and SGD

Cited by in corpus (7)

• Stronger NAS with Weaker Predictors
• Few-shot Neural Architecture Search
• Evolving Search Space for Neural Architecture Search
• How Does Supernet Help in Neural Architecture Search?
• Neural Architecture Search with Random Labels
• BN-NAS: Neural Architecture Search with Batch Normalization
• One-Shot Neural Ensemble Architecture Search by Diversity-Guided Search Space Shrinking