Best Practices for Scientific Research on Neural Architecture Search

Abstract

Finding a well-performing architecture is often tedious for both DL practitioners and researchers, leading to tremendous interest in the automation of this task by means of neural architecture search (NAS). Although the community has made major strides in developing better NAS methods, the quality of scientific empirical evaluations in the young field of NAS is still lacking behind that of other areas of machine learning. To address this issue, we describe a set of possible issues and ways to avoid them, leading to the NAS best practices checklist available at http://automl.org/nas_checklist.pdf.

References in corpus (20)

• Decoupled Weight Decay Regularization
• Neural Architecture Search with Reinforcement Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• DARTS: Differentiable Architecture Search
• AutoAugment: Learning Augmentation Policies from Data
• Searching for Efficient Multi-Scale Architectures for Dense Image Prediction
• Deep Reinforcement Learning that Matters
• Shake-Shake regularization
• Understanding and Robustifying Differentiable Architecture Search
• NAS-Bench-101: Towards Reproducible Neural Architecture Search
• The Evolved Transformer
• Evaluating the Search Phase of Neural Architecture Search
• AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data
• DeepArchitect: Automatically Designing and Training Deep Architectures
• NAS evaluation is frustratingly hard
• Exploring Randomly Wired Neural Networks for Image Recognition
• Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization
• Evaluating Efficient Performance Estimators of Neural Architectures
• Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL
• Learning to Design RNA

Cited by in corpus (23)

• Understanding and Robustifying Differentiable Architecture Search
• Neuroevolution in Deep Neural Networks: Current Trends and Future Challenges
• NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size
• NAS evaluation is frustratingly hard
• NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
• Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Evaluating Generic Auto-ML Tools for Computational Pathology
• Differential Evolution for Neural Architecture Search
• How Powerful are Performance Predictors in Neural Architecture Search?
• Can weight sharing outperform random architecture search? An investigation with TuNAS
• An Introduction to Neural Architecture Search for Convolutional Networks
• ImmuNeCS: Neural Committee Search by an Artificial Immune System
• ScaleNAS: One-Shot Learning of Scale-Aware Representations for Visual Recognition
• Fine-Grained Stochastic Architecture Search
• A Study on Encodings for Neural Architecture Search
• FEAR: A Simple Lightweight Method to Rank Architectures
• RandomNet: Towards Fully Automatic Neural Architecture Design for Multimodal Learning
• Hyperparameter optimization with REINFORCE and Transformers
• Exploring the Loss Landscape in Neural Architecture Search
• Improving the sample-efficiency of neural architecture search with reinforcement learning
• GPNAS: A Neural Network Architecture Search Framework Based on Graphical Predictor
• Mutation is all you need