Sample-Efficient Neural Architecture Search by Learning Action Space

Abstract

Neural Architecture Search (NAS) has emerged as a promising technique for automatic neural network design. However, existing MCTS based NAS approaches often utilize manually designed action space, which is not directly related to the performance metric to be optimized (e.g., accuracy), leading to sample-inefficient explorations of architectures. To improve the sample efficiency, this paper proposes Latent Action Neural Architecture Search (LaNAS), which learns actions to recursively partition the search space into good or bad regions that contain networks with similar performance metrics. During the search phase, as different action sequences lead to regions with different performance, the search efficiency can be significantly improved by biasing towards the good regions. On three NAS tasks, empirical results demonstrate that LaNAS is at least an order more sample efficient than baseline methods including evolutionary algorithms, Bayesian optimizations, and random search. When applied in practice, both one-shot and regular LaNAS consistently outperform existing results. Particularly, LaNAS achieves 99.0% accuracy on CIFAR-10 and 80.8% top1 accuracy at 600 MFLOPS on ImageNet in only 800 samples, significantly outperforming AmoebaNet with 33x fewer samples. Our code is publicly available at https://github.com/facebookresearch/LaMCTS.

arXiv commentAccepted at TPAMI-2021

References in corpus (25)

• Practical Bayesian Optimization of Machine Learning Algorithms
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Neural Architecture Search with Reinforcement Learning
• Neural Architecture Optimization
• PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
• NAS-Bench-101: Towards Reproducible Neural Architecture Search
• Random Search and Reproducibility for Neural Architecture Search
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• The Evolved Transformer
• DetNAS: Backbone Search for Object Detection
• Evaluating the Search Phase of Neural Architecture Search
• Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation
• DeepArchitect: Automatically Designing and Training Deep Architectures
• FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search
• Exploring Randomly Wired Neural Networks for Image Recognition
• Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS
• Bayesian Multi-Scale Optimistic Optimization
• AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search
• Adaptive Stochastic Natural Gradient Method for One-Shot Neural Architecture Search
• ELF OpenGo: An Analysis and Open Reimplementation of AlphaZero
• Breaking the Curse of Space Explosion: Towards Efficient NAS with Curriculum Search
• FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining
• Neural Architecture Search using Deep Neural Networks and Monte Carlo Tree Search
• Few-shot Neural Architecture Search
• Solving Black-Box Optimization Challenge via Learning Search Space Partition for Local Bayesian Optimization

Cited by in corpus (18)

• BRP-NAS: Prediction-based NAS using GCNs
• Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS
• NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Search
• Learning Search Space Partition for Black-box Optimization using Monte Carlo Tree Search
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Interpretable Neural Architecture Search via Bayesian Optimisation with Weisfeiler-Lehman Kernels
• Few-shot Neural Architecture Search
• CATE: Computation-aware Neural Architecture Encoding with Transformers
• Robust Learning Meets Generative Models: Can Proxy Distributions Improve Adversarial Robustness?
• Accuracy Prediction with Non-neural Model for Neural Architecture Search
• Solving Black-Box Optimization Challenge via Learning Search Space Partition for Local Bayesian Optimization
• Multi-objective Optimization by Learning Space Partitions
• HAO: Hardware-aware neural Architecture Optimization for Efficient Inference
• Contrastive Embeddings for Neural Architectures
• Prioritized Architecture Sampling with Monto-Carlo Tree Search
• Pi-NAS: Improving Neural Architecture Search by Reducing Supernet Training Consistency Shift
• ISyNet: Convolutional Neural Networks design for AI accelerator
• EfficientBERT: Progressively Searching Multilayer Perceptron via Warm-up Knowledge Distillation