Semi-Supervised Neural Architecture Search

Abstract

Neural architecture search (NAS) relies on a good controller to generate better architectures or predict the accuracy of given architectures. However, training the controller requires both abundant and high-quality pairs of architectures and their accuracy, while it is costly to evaluate an architecture and obtain its accuracy. In this paper, we propose SemiNAS, a semi-supervised NAS approach that leverages numerous unlabeled architectures (without evaluation and thus nearly no cost). Specifically, SemiNAS 1) trains an initial accuracy predictor with a small set of architecture-accuracy data pairs; 2) uses the trained accuracy predictor to predict the accuracy of large amount of architectures (without evaluation); and 3) adds the generated data pairs to the original data to further improve the predictor. The trained accuracy predictor can be applied to various NAS algorithms by predicting the accuracy of candidate architectures for them. SemiNAS has two advantages: 1) It reduces the computational cost under the same accuracy guarantee. On NASBench-101 benchmark dataset, it achieves comparable accuracy with gradient-based method while using only 1/7 architecture-accuracy pairs. 2) It achieves higher accuracy under the same computational cost. It achieves 94.02% test accuracy on NASBench-101, outperforming all the baselines when using the same number of architectures. On ImageNet, it achieves 23.5% top-1 error rate (under 600M FLOPS constraint) using 4 GPU-days for search. We further apply it to LJSpeech text to speech task and it achieves 97% intelligibility rate in the low-resource setting and 15% test error rate in the robustness setting, with 9%, 7% improvements over the baseline respectively.

arXiv commentNeurIPS 2020

References in corpus (18)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Neural Architecture Search with Reinforcement Learning
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Unsupervised Data Augmentation for Consistency Training
• Neural Architecture Optimization
• SNAS: Stochastic Neural Architecture Search
• NAS-Bench-101: Towards Reproducible Neural Architecture Search
• Random Search and Reproducibility for Neural Architecture Search
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• The Evolved Transformer
• Evaluating the Search Phase of Neural Architecture Search
• Progressive Neural Architecture Search
• Tacotron: Towards End-to-End Speech Synthesis
• Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation
• Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning
• Almost Unsupervised Text to Speech and Automatic Speech Recognition
• Neural Predictor for Neural Architecture Search

Cited by in corpus (18)

• AutoML: A Survey of the State-of-the-Art
• Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• EfficientTDNN: Efficient Architecture Search for Speaker Recognition
• How Powerful are Performance Predictors in Neural Architecture Search?
• Stronger NAS with Weaker Predictors
• Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognition
• CATE: Computation-aware Neural Architecture Encoding with Transformers
• Accuracy Prediction with Non-neural Model for Neural Architecture Search
• GradSign: Model Performance Inference with Theoretical Insights
• Contrastive Neural Architecture Search with Neural Architecture Comparators
• FNAS: Uncertainty-Aware Fast Neural Architecture Search
• GENNAPE: Towards Generalized Neural Architecture Performance Estimators
• A General-Purpose Transferable Predictor for Neural Architecture Search
• LightSpeech: Lightweight and Fast Text to Speech with Neural Architecture Search
• AIO-P: Expanding Neural Performance Predictors Beyond Image Classification
• Pretraining Neural Architecture Search Controllers with Locality-based Self-Supervised Learning
• Training BatchNorm Only in Neural Architecture Search and Beyond