NAS-Bench-101: Towards Reproducible Neural Architecture Search
arXiv:1902.09635
Abstract
Recent advances in neural architecture search (NAS) demand tremendous computational resources, which makes it difficult to reproduce experiments and imposes a barrier-to-entry to researchers without access to large-scale computation. We aim to ameliorate these problems by introducing NAS-Bench-101, the first public architecture dataset for NAS research. To build NAS-Bench-101, we carefully constructed a compact, yet expressive, search space, exploiting graph isomorphisms to identify 423k unique convolutional architectures. We trained and evaluated all of these architectures multiple times on CIFAR-10 and compiled the results into a large dataset of over 5 million trained models. This allows researchers to evaluate the quality of a diverse range of models in milliseconds by querying the pre-computed dataset. We demonstrate its utility by analyzing the dataset as a whole and by benchmarking a range of architecture optimization algorithms.
Published in the Proceedings of the 36th International Conference on Machine Learning
Cited by in corpus (67)
- A Survey on Evolutionary Neural Architecture Search
- NAS-FAS: Static-Dynamic Central Difference Network Search for Face Anti-Spoofing
- NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size
- Automated Machine Learning on Graphs: A Survey
- EEEA-Net: An Early Exit Evolutionary Neural Architecture Search
- A Comprehensive Survey on Hardware-Aware Neural Architecture Search
- Towards Green Automated Machine Learning: Status Quo and Future Directions
- EPE-NAS: Efficient Performance Estimation Without Training for Neural Architecture Search
- Contrastive Self-supervised Neural Architecture Search
- Deeper Insights into Weight Sharing in Neural Architecture Search
- Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization
- Evaluating Efficient Performance Estimators of Neural Architectures
- Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
- EvoPose2D: Pushing the Boundaries of 2D Human Pose Estimation using Accelerated Neuroevolution with Weight Transfer
- Differential Evolution for Neural Architecture Search
- Neural Predictor for Neural Architecture Search
- ANNETTE: Accurate Neural Network Execution Time Estimation with Stacked Models
- Towards NNGP-guided Neural Architecture Search
- Stronger NAS with Weaker Predictors
- AttentiveNAS: Improving Neural Architecture Search via Attentive Sampling
- Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognition
- NASGEM: Neural Architecture Search via Graph Embedding Method
- Neural Architecture Optimization with Graph VAE
- Techniques for Automated Machine Learning
- Standing on the Shoulders of Giants: Hardware and Neural Architecture Co-Search with Hot Start
- An Introduction to Neural Architecture Search for Convolutional Networks
- Efficient Forward Architecture Search
- Computational catalyst discovery: Active classification through myopic multiscale sampling
- Learning Versatile Neural Architectures by Propagating Network Codes
- CATE: Computation-aware Neural Architecture Encoding with Transformers
- Self-supervised Representation Learning for Evolutionary Neural Architecture Search
- GENNAPE: Towards Generalized Neural Architecture Performance Estimators
- MUXConv: Information Multiplexing in Convolutional Neural Networks
- Contrastive Neural Architecture Search with Neural Architecture Comparators
- EH-DNAS: End-to-End Hardware-aware Differentiable Neural Architecture Search
- Searching for Stage-wise Neural Graphs In the Limit
- Fine-Grained Stochastic Architecture Search
- TransNAS-Bench-101: Improving Transferability and Generalizability of Cross-Task Neural Architecture Search
- Bag of Tricks for Neural Architecture Search
- Binarized Neural Architecture Search for Efficient Object Recognition
- Near-linear Time Gaussian Process Optimization with Adaptive Batching and Resparsification
- NAS-HPO-Bench-II: A Benchmark Dataset on Joint Optimization of Convolutional Neural Network Architecture and Training Hyperparameters
- AIO-P: Expanding Neural Performance Predictors Beyond Image Classification
- Binarized Neural Architecture Search
- Generic Neural Architecture Search via Regression
- Transfer NAS: Knowledge Transfer between Search Spaces with Transformer Agents
- Fitness Landscape Footprint: A Framework to Compare Neural Architecture Search Problems
- FEAR: A Simple Lightweight Method to Rank Architectures
- FixNorm: Dissecting Weight Decay for Training Deep Neural Networks
- Rethinking Neural Operations for Diverse Tasks
- ProxyBO: Accelerating Neural Architecture Search via Bayesian Optimization with Zero-cost Proxies
- Lessons from the Clustering Analysis of a Search Space: A Centroid-based Approach to Initializing NAS
- BaLeNAS: Differentiable Architecture Search via the Bayesian Learning Rule
- Efficient Sampling for Predictor-Based Neural Architecture Search
- Efficient Model Performance Estimation via Feature Histories
- Pretraining Neural Architecture Search Controllers with Locality-based Self-Supervised Learning
- Disentangled Neural Architecture Search
- Joint Learning of Neural Transfer and Architecture Adaptation for Image Recognition
- Enhanced Gradient for Differentiable Architecture Search
- Neural networks adapting to datasets: learning network size and topology
- Connection Sensitivity Matters for Training-free DARTS: From Architecture-Level Scoring to Operation-Level Sensitivity Analysis
- An Analysis of Super-Net Heuristics in Weight-Sharing NAS
- Training BatchNorm Only in Neural Architecture Search and Beyond
- LETI: Latency Estimation Tool and Investigation of Neural Networks inference on Mobile GPU
- Accelerating Neural Architecture Search via Proxy Data
- Mutation is all you need
- GPNAS: A Neural Network Architecture Search Framework Based on Graphical Predictor