Evaluating the Search Phase of Neural Architecture Search

Abstract

Neural Architecture Search (NAS) aims to facilitate the design of deep networks for new tasks. Existing techniques rely on two stages: searching over the architecture space and validating the best architecture. NAS algorithms are currently compared solely based on their results on the downstream task. While intuitive, this fails to explicitly evaluate the effectiveness of their search strategies. In this paper, we propose to evaluate the NAS search phase. To this end, we compare the quality of the solutions obtained by NAS search policies with that of random architecture selection. We find that: (i) On average, the state-of-the-art NAS algorithms perform similarly to the random policy; (ii) the widely-used weight sharing strategy degrades the ranking of the NAS candidates to the point of not reflecting their true performance, thus reducing the effectiveness of the search process. We believe that our evaluation framework will be key to designing NAS strategies that consistently discover architectures superior to random ones.

We find that random policy in NAS works amazingly well and propose an evaluation framework to have a fair comparison. Adding additional results on standard CNN search space used for weight sharing and NASBench-101. 8 pages

References in corpus (13)

• Neural Architecture Search with Reinforcement Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• Regularizing and Optimizing LSTM Language Models
• Efficient Architecture Search by Network Transformation
• Random Search and Reproducibility for Neural Architecture Search
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• The Evolved Transformer
• DetNAS: Backbone Search for Object Detection
• Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation
• FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search
• BayesNAS: A Bayesian Approach for Neural Architecture Search
• Exploring Randomly Wired Neural Networks for Image Recognition
• AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search

Cited by in corpus (50)

• AutoML: A Survey of the State-of-the-Art
• A Survey on Evolutionary Neural Architecture Search
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• Neural Architecture Transfer
• NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size
• Best Practices for Scientific Research on Neural Architecture Search
• Exploring Randomly Wired Neural Networks for Image Recognition
• Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS
• AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search
• Semi-Supervised Neural Architecture Search
• Stabilizing DARTS with Amended Gradient Estimation on Architectural Parameters
• Contrastive Self-supervised Neural Architecture Search
• Sample-Efficient Neural Architecture Search by Learning Action Space
• Understanding Neural Architecture Search Techniques
• EvoPose2D: Pushing the Boundaries of 2D Human Pose Estimation using Accelerated Neuroevolution with Weight Transfer
• DSNAS: Direct Neural Architecture Search without Parameter Retraining
• A Generic Graph-based Neural Architecture Encoding Scheme for Predictor-based NAS
• XNAS: Neural Architecture Search with Expert Advice
• HARK Side of Deep Learning -- From Grad Student Descent to Automated Machine Learning
• How Powerful are Performance Predictors in Neural Architecture Search?
• ReNAS:Relativistic Evaluation of Neural Architecture Search
• One-Shot Neural Architecture Search via Compressive Sensing
• BlockSwap: Fisher-guided Block Substitution for Network Compression on a Budget
• SGAS: Sequential Greedy Architecture Search
• BETANAS: BalancEd TrAining and selective drop for Neural Architecture Search
• Core-set Sampling for Efficient Neural Architecture Search
• Optimal Transport Kernels for Sequential and Parallel Neural Architecture Search
• Understanding Architectures Learnt by Cell-based Neural Architecture Search
• Neural Architecture Search using Property Guided Synthesis
• ImmuNeCS: Neural Committee Search by an Artificial Immune System
• AMEIR: Automatic Behavior Modeling, Interaction Exploration and MLP Investigation in the Recommender System
• Neural Architecture Search by Estimation of Network Structure Distributions
• HM-NAS: Efficient Neural Architecture Search via Hierarchical Masking
• Neural Architecture Search Over a Graph Search Space
• FNA++: Fast Network Adaptation via Parameter Remapping and Architecture Search
• PONAS: Progressive One-shot Neural Architecture Search for Very Efficient Deployment
• DC-NAS: Divide-and-Conquer Neural Architecture Search
• Fine-Grained Stochastic Architecture Search
• How Does Supernet Help in Neural Architecture Search?
• Fisher Task Distance and Its Application in Neural Architecture Search
• ResNetX: a more disordered and deeper network architecture
• Multi-objective Optimization by Learning Space Partitions
• Generic Neural Architecture Search via Regression
• Robustifying DARTS by Eliminating Information Bypass Leakage via Explicit Sparse Regularization
• Neural Architecture Search with Random Labels
• Landmark Regularization: Ranking Guided Super-Net Training in Neural Architecture Search
• NeuralArTS: Structuring Neural Architecture Search with Type Theory
• AutoBSS: An Efficient Algorithm for Block Stacking Style Search
• Poisoning the Search Space in Neural Architecture Search
• Self-Constructing Neural Networks Through Random Mutation