NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size

Abstract

Neural architecture search (NAS) has attracted a lot of attention and has been illustrated to bring tangible benefits in a large number of applications in the past few years. Architecture topology and architecture size have been regarded as two of the most important aspects for the performance of deep learning models and the community has spawned lots of searching algorithms for both aspects of the neural architectures. However, the performance gain from these searching algorithms is achieved under different search spaces and training setups. This makes the overall performance of the algorithms to some extent incomparable and the improvement from a sub-module of the searching model unclear. In this paper, we propose NATS-Bench, a unified benchmark on searching for both topology and size, for (almost) any up-to-date NAS algorithm. NATS-Bench includes the search space of 15,625 neural cell candidates for architecture topology and 32,768 for architecture size on three datasets. We analyze the validity of our benchmark in terms of various criteria and performance comparison of all candidates in the search space. We also show the versatility of NATS-Bench by benchmarking 13 recent state-of-the-art NAS algorithms on it. All logs and diagnostic information trained using the same setup for each candidate are provided. This facilitates a much larger community of researchers to focus on developing better NAS algorithms in a more comparable and computationally cost friendly environment. All codes are publicly available at: https://xuanyidong.com/assets/projects/NATS-Bench.

arXiv commentAccepted to IEEE TPAMI 2021, an extended version of NAS-Bench-201 (ICLR 2020) [arXiv:2001.00326]

References in corpus (20)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• mixup: Beyond Empirical Risk Minimization
• Neural Architecture Search with Reinforcement Learning
• DARTS: Differentiable Architecture Search
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• DropBlock: A regularization method for convolutional networks
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
• Efficient Architecture Search by Network Transformation
• NAS-Bench-101: Towards Reproducible Neural Architecture Search
• One-Shot Neural Architecture Search via Self-Evaluated Template Network
• Evaluating the Search Phase of Neural Architecture Search
• Network Pruning via Transformable Architecture Search
• Graph HyperNetworks for Neural Architecture Search
• Best Practices for Scientific Research on Neural Architecture Search
• NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
• Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization
• AutoHAS: Efficient Hyperparameter and Architecture Search
• PyGlove: Symbolic Programming for Automated Machine Learning

Cited by in corpus (18)

• A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
• A Comprehensive Survey on Hardware-Aware Neural Architecture Search
• Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective
• Neural Architecture Search without Training
• FreeREA: Training-Free Evolution-based Architecture Search
• HW-NAS-Bench:Hardware-Aware Neural Architecture Search Benchmark
• Optimizing Convolutional Neural Network Architecture
• Benchmarking optimization algorithms for auto-tuning GPU kernels
• Isometric Propagation Network for Generalized Zero-shot Learning
• RBFleX-NAS: Training-Free Neural Architecture Search Using Radial Basis Function Kernel and Hyperparameter Detection
• RARTS: An Efficient First-Order Relaxed Architecture Search Method
• EH-DNAS: End-to-End Hardware-aware Differentiable Neural Architecture Search
• FEAR: A Simple Lightweight Method to Rank Architectures
• NAS-HPO-Bench-II: A Benchmark Dataset on Joint Optimization of Convolutional Neural Network Architecture and Training Hyperparameters
• Contrastive Embeddings for Neural Architectures
• Mixed Variable Bayesian Optimization with Frequency Modulated Kernels
• Accelerating Neural Architecture Search via Proxy Data
• Tetra-AML: Automatic Machine Learning via Tensor Networks