Random Search and Reproducibility for Neural Architecture Search

Abstract

Neural architecture search (NAS) is a promising research direction that has the potential to replace expert-designed networks with learned, task-specific architectures. In this work, in order to help ground the empirical results in this field, we propose new NAS baselines that build off the following observations: (i) NAS is a specialized hyperparameter optimization problem; and (ii) random search is a competitive baseline for hyperparameter optimization. Leveraging these observations, we evaluate both random search with early-stopping and a novel random search with weight-sharing algorithm on two standard NAS benchmarks---PTB and CIFAR-10. Our results show that random search with early-stopping is a competitive NAS baseline, e.g., it performs at least as well as ENAS, a leading NAS method, on both benchmarks. Additionally, random search with weight-sharing outperforms random search with early-stopping, achieving a state-of-the-art NAS result on PTB and a highly competitive result on CIFAR-10. Finally, we explore the existing reproducibility issues of published NAS results. We note the lack of source material needed to exactly reproduce these results, and further discuss the robustness of published results given the various sources of variability in NAS experimental setups. Relatedly, we provide all information (code, random seeds, documentation) needed to exactly reproduce our results, and report our random search with weight-sharing results for each benchmark on multiple runs.

arXiv commentV2 Changelog: - Modified footnote 2 for ENAS. - Expanded broad reproducibility study for random search with WS for CNN to 6 sets of random seeds v3 Changelog: - Added journal reference - Updated acknowledgements

References in corpus (13)

• Practical Bayesian Optimization of Machine Learning Algorithms
• Neural Architecture Search with Reinforcement Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• Neural Architecture Search: A Survey
• DARTS: Differentiable Architecture Search
• Efficient Neural Architecture Search via Parameter Sharing
• Large-Scale Evolution of Image Classifiers
• Neural Architecture Optimization
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• Path-Level Network Transformation for Efficient Architecture Search
• DeepArchitect: Automatically Designing and Training Deep Architectures
• Multi-fidelity Bayesian Optimisation with Continuous Approximations
• Learnable Embedding Space for Efficient Neural Architecture Compression

Cited by in corpus (103)

• Neural Architecture Search: A Survey
• Understanding and Robustifying Differentiable Architecture Search
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• A Survey on Neural Architecture Search
• A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
• Neural Architecture Transfer
• DetNAS: Backbone Search for Object Detection
• Evaluating the Search Phase of Neural Architecture Search
• AutoML-Zero: Evolving Machine Learning Algorithms From Scratch
• NAS evaluation is frustratingly hard
• Neural Predictor based Quantum Architecture Search
• Exploring Randomly Wired Neural Networks for Image Recognition
• Stabilizing Differentiable Architecture Search via Perturbation-based Regularization
• NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
• A System for Massively Parallel Hyperparameter Tuning
• Evolving Normalization-Activation Layers
• Semi-Supervised Neural Architecture Search
• AutoGAN: Neural Architecture Search for Generative Adversarial Networks
• Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonfly
• Stabilizing DARTS with Amended Gradient Estimation on Architectural Parameters
• DAIS: Automatic Channel Pruning via Differentiable Annealing Indicator Search
• Deeper Insights into Weight Sharing in Neural Architecture Search
• Sample-Efficient Neural Architecture Search by Learning Action Space
• GOLD-NAS: Gradual, One-Level, Differentiable
• Understanding Neural Architecture Search Techniques
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• DSNAS: Direct Neural Architecture Search without Parameter Retraining
• How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAS
• XNAS: Neural Architecture Search with Expert Advice
• Neural Predictor for Neural Architecture Search
• MANAS: Multi-Agent Neural Architecture Search
• HARK Side of Deep Learning -- From Grad Student Descent to Automated Machine Learning
• How Powerful are Performance Predictors in Neural Architecture Search?
• Interpretable Neural Architecture Search via Bayesian Optimisation with Weisfeiler-Lehman Kernels
• Multi-Objective Evolutionary Design of Deep Convolutional Neural Networks for Image Classification
• One-Shot Neural Architecture Search via Compressive Sensing
• NSGANetV2: Evolutionary Multi-Objective Surrogate-Assisted Neural Architecture Search
• AutoOD: Automated Outlier Detection via Curiosity-guided Search and Self-imitation Learning
• Evolutionary-Neural Hybrid Agents for Architecture Search
• Learning Causal Models Online
• To Share or Not To Share: A Comprehensive Appraisal of Weight-Sharing
• BlockSwap: Fisher-guided Block Substitution for Network Compression on a Budget
• Densely Connected Search Space for More Flexible Neural Architecture Search
• AdversarialNAS: Adversarial Neural Architecture Search for GANs
• Pose Neural Fabrics Search
• GreedyNAS: Towards Fast One-Shot NAS with Greedy Supernet
• Feature Partitioning for Efficient Multi-Task Architectures
• A Semi-Supervised Assessor of Neural Architectures
• NASGEM: Neural Architecture Search via Graph Embedding Method
• The Design and Implementation of a Scalable DL Benchmarking Platform
• Efficient Forward Architecture Search
• AlphaGAN: Fully Differentiable Architecture Search for Generative Adversarial Networks
• MnasFPN: Learning Latency-aware Pyramid Architecture for Object Detection on Mobile Devices
• Hierarchical Neural Architecture Search via Operator Clustering
• Balanced One-shot Neural Architecture Optimization
• Towards Improving the Consistency, Efficiency, and Flexibility of Differentiable Neural Architecture Search
• ImmuNeCS: Neural Committee Search by an Artificial Immune System
• Simplifying Architecture Search for Graph Neural Network
• A Study of the Learning Progress in Neural Architecture Search Techniques
• S3NAS: Fast NPU-aware Neural Architecture Search Methodology
• VINNAS: Variational Inference-based Neural Network Architecture Search
• Accuracy Prediction with Non-neural Model for Neural Architecture Search
• HM-NAS: Efficient Neural Architecture Search via Hierarchical Masking
• Learning Architectures for Binary Networks
• FNAS: Uncertainty-Aware Fast Neural Architecture Search
• DC-NAS: Divide-and-Conquer Neural Architecture Search
• A Study on Encodings for Neural Architecture Search
• Efficient hyperparameter optimization by way of PAC-Bayes bound minimization
• Quantity vs. Quality: On Hyperparameter Optimization for Deep Reinforcement Learning
• Single-Path Mobile AutoML: Efficient ConvNet Design and NAS Hyperparameter Optimization
• TextNAS: A Neural Architecture Search Space tailored for Text Representation
• Rethinking Performance Estimation in Neural Architecture Search
• AutoSTR: Efficient Backbone Search for Scene Text Recognition
• Bag of Tricks for Neural Architecture Search
• Efficient Relation-aware Scoring Function Search for Knowledge Graph Embedding
• Transfer NAS: Knowledge Transfer between Search Spaces with Transformer Agents
• Efficient Neural Architecture Search for End-to-end Speech Recognition via Straight-Through Gradients
• NetAdaptV2: Efficient Neural Architecture Search with Fast Super-Network Training and Architecture Optimization
• A Design Space Study for LISTA and Beyond
• ResNetX: a more disordered and deeper network architecture
• Intelligence plays dice: Stochasticity is essential for machine learning
• RAPDARTS: Resource-Aware Progressive Differentiable Architecture Search
• Bonsai-Net: One-Shot Neural Architecture Search via Differentiable Pruners
• Hyperparameter optimization with REINFORCE and Transformers
• Exploring the Loss Landscape in Neural Architecture Search
• BNAS v2: Learning Architectures for Binary Networks with Empirical Improvements
• Generalizing Tensor Decomposition for N-ary Relational Knowledge Bases
• ElixirNet: Relation-aware Network Architecture Adaptation for Medical Lesion Detection
• Direct Federated Neural Architecture Search
• Knowledge Transfer Graph for Deep Collaborative Learning
• Conceptual Expansion Neural Architecture Search (CENAS)
• Hyperparameter Optimization in Neural Networks via Structured Sparse Recovery
• Real-Time Semantic Segmentation via Auto Depth, Downsampling Joint Decision and Feature Aggregation
• Efficient Novelty-Driven Neural Architecture Search
• EfficientPose: Efficient Human Pose Estimation with Neural Architecture Search
• Revisiting Neural Architecture Search
• Neural Architecture Search in operational context: a remote sensing case-study
• AutoBSS: An Efficient Algorithm for Block Stacking Style Search
• Efficient Data-specific Model Search for Collaborative Filtering
• Connection Sensitivity Matters for Training-free DARTS: From Architecture-Level Scoring to Operation-Level Sensitivity Analysis
• Shape Adaptor: A Learnable Resizing Module
• Memory-Efficient Hierarchical Neural Architecture Search for Image Restoration
• Evolving Neural Architecture Using One Shot Model