AutoHAS: Efficient Hyperparameter and Architecture Search

Abstract

Efficient hyperparameter or architecture search methods have shown remarkable results, but each of them is only applicable to searching for either hyperparameters (HPs) or architectures. In this work, we propose a unified pipeline, AutoHAS, to efficiently search for both architectures and hyperparameters. AutoHAS learns to alternately update the shared network weights and a reinforcement learning (RL) controller, which learns the probability distribution for the architecture candidates and HP candidates. A temporary weight is introduced to store the updated weight from the selected HPs (by the controller), and a validation accuracy based on this temporary weight serves as a reward to update the controller. In experiments, we show AutoHAS is efficient and generalizable to different search spaces, baselines and datasets. In particular, AutoHAS can improve the accuracy over popular network architectures, such as ResNet and EfficientNet, on CIFAR-10/100, ImageNet, and four more other datasets.

arXiv commentAccepted to 2nd Workshop on Neural Architecture Search at ICLR 2021

References in corpus (11)

• Neural Architecture Search with Reinforcement Learning
• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• DARTS: Differentiable Architecture Search
• Efficient Neural Architecture Search via Parameter Sharing
• Scalable Bayesian Optimization Using Deep Neural Networks
• Designing Neural Network Architectures using Reinforcement Learning
• Understanding and Robustifying Differentiable Architecture Search
• Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search
• Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization
• FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions
• Can weight sharing outperform random architecture search? An investigation with TuNAS

Cited by in corpus (4)

• NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size
• Generalization Guarantees for Neural Architecture Search with Train-Validation Split
• NAS-HPO-Bench-II: A Benchmark Dataset on Joint Optimization of Convolutional Neural Network Architecture and Training Hyperparameters
• Mixed Variable Bayesian Optimization with Frequency Modulated Kernels