NAS evaluation is frustratingly hard

Abstract

Neural Architecture Search (NAS) is an exciting new field which promises to be as much as a game-changer as Convolutional Neural Networks were in 2012. Despite many great works leading to substantial improvements on a variety of tasks, comparison between different methods is still very much an open issue. While most algorithms are tested on the same datasets, there is no shared experimental protocol followed by all. As such, and due to the under-use of ablation studies, there is a lack of clarity regarding why certain methods are more effective than others. Our first contribution is a benchmark of $8$ NAS methods on $5$ datasets. To overcome the hurdle of comparing methods with different search spaces, we propose using a method's relative improvement over the randomly sampled average architecture, which effectively removes advantages arising from expertly engineered search spaces or training protocols. Surprisingly, we find that many NAS techniques struggle to significantly beat the average architecture baseline. We perform further experiments with the commonly used DARTS search space in order to understand the contribution of each component in the NAS pipeline. These experiments highlight that: (i) the use of tricks in the evaluation protocol has a predominant impact on the reported performance of architectures; (ii) the cell-based search space has a very narrow accuracy range, such that the seed has a considerable impact on architecture rankings; (iii) the hand-designed macro-structure (cells) is more important than the searched micro-structure (operations); and (iv) the depth-gap is a real phenomenon, evidenced by the change in rankings between $8$ and $20$ cell architectures. To conclude, we suggest best practices, that we hope will prove useful for the community and help mitigate current NAS pitfalls. The code used is available at https://github.com/antoyang/NAS-Benchmark.

Published as a conference paper at ICLR2020; 13 pages; 10 figures

References in corpus (11)

• Improved Regularization of Convolutional Neural Networks with Cutout
• DARTS: Differentiable Architecture Search
• AutoAugment: Learning Augmentation Policies from Data
• Large-Scale Evolution of Image Classifiers
• Neural Architecture Optimization
• Random Search and Reproducibility for Neural Architecture Search
• Best Practices for Scientific Research on Neural Architecture Search
• sharpDARTS: Faster and More Accurate Differentiable Architecture Search
• XNAS: Neural Architecture Search with Expert Advice
• HARK Side of Deep Learning -- From Grad Student Descent to Automated Machine Learning
• MANAS: Multi-Agent Neural Architecture Search

Cited by in corpus (51)

• AutoML: A Survey of the State-of-the-Art
• NAS-FAS: Static-Dynamic Central Difference Network Search for Face Anti-Spoofing
• DARTS+: Improved Differentiable Architecture Search with Early Stopping
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
• Best Practices for Scientific Research on Neural Architecture Search
• Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS
• EPE-NAS: Efficient Performance Estimation Without Training for Neural Architecture Search
• ISTA-NAS: Efficient and Consistent Neural Architecture Search by Sparse Coding
• DARTS-: Robustly Stepping out of Performance Collapse Without Indicators
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Blockwisely Supervised Neural Architecture Search with Knowledge Distillation
• HW-NAS-Bench:Hardware-Aware Neural Architecture Search Benchmark
• How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAS
• Neural Ensemble Search for Uncertainty Estimation and Dataset Shift
• MANAS: Multi-Agent Neural Architecture Search
• How Powerful are Performance Predictors in Neural Architecture Search?
• Neural Architecture Generator Optimization
• Speedy Performance Estimation for Neural Architecture Search
• To Share or Not To Share: A Comprehensive Appraisal of Weight-Sharing
• Less is More: Proxy Datasets in NAS approaches
• Cyclic Differentiable Architecture Search
• Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL
• DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecture
• DrNAS: Dirichlet Neural Architecture Search
• Accuracy Prediction with Non-neural Model for Neural Architecture Search
• HourNAS: Extremely Fast Neural Architecture Search Through an Hourglass Lens
• Evolving Search Space for Neural Architecture Search
• Multi-objective Neural Architecture Search via Non-stationary Policy Gradient
• Contrastive Neural Architecture Search with Neural Architecture Comparators
• The Nonlinearity Coefficient - A Practical Guide to Neural Architecture Design
• Bag of Tricks for Neural Architecture Search
• TransNAS-Bench-101: Improving Transferability and Generalizability of Cross-Task Neural Architecture Search
• A Study on Encodings for Neural Architecture Search
• FNA++: Fast Network Adaptation via Parameter Remapping and Architecture Search
• How Does Supernet Help in Neural Architecture Search?
• Optimizing Neural Architecture Search using Limited GPU Time in a Dynamic Search Space: A Gene Expression Programming Approach
• Neighborhood-Aware Neural Architecture Search
• Fitness Landscape Footprint: A Framework to Compare Neural Architecture Search Problems
• Heed the Noise in Performance Evaluations in Neural Architecture Search
• NAS-HPO-Bench-II: A Benchmark Dataset on Joint Optimization of Convolutional Neural Network Architecture and Training Hyperparameters
• Rethinking Neural Operations for Diverse Tasks
• Robustifying DARTS by Eliminating Information Bypass Leakage via Explicit Sparse Regularization
• Raw Differentiable Architecture Search for Speech Deepfake and Spoofing Detection
• An Analysis of Super-Net Heuristics in Weight-Sharing NAS
• Revisiting Neural Architecture Search
• Joint Learning of Neural Transfer and Architecture Adaptation for Image Recognition
• AutoBSS: An Efficient Algorithm for Block Stacking Style Search
• AUTOKD: Automatic Knowledge Distillation Into A Student Architecture Family
• Guided Evolution for Neural Architecture Search
• AdvantageNAS: Efficient Neural Architecture Search with Credit Assignment