Can weight sharing outperform random architecture search? An investigation with TuNAS

Abstract

Efficient Neural Architecture Search methods based on weight sharing have shown good promise in democratizing Neural Architecture Search for computer vision models. There is, however, an ongoing debate whether these efficient methods are significantly better than random search. Here we perform a thorough comparison between efficient and random search methods on a family of progressively larger and more challenging search spaces for image classification and detection on ImageNet and COCO. While the efficacies of both methods are problem-dependent, our experiments demonstrate that there are large, realistic tasks where efficient search methods can provide substantial gains over random search. In addition, we propose and evaluate techniques which improve the quality of searched architectures and reduce the need for manual hyper-parameter tuning. Source code and experiment data are available at https://github.com/google-research/google-research/tree/master/tunas

arXiv commentPublished at CVPR 2020

References in corpus (15)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Gaussian Error Linear Units (GELUs)
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• SGDR: Stochastic Gradient Descent with Warm Restarts
• DARTS: Differentiable Architecture Search
• Searching for Activation Functions
• Large-Scale Evolution of Image Classifiers
• Neural Architecture Optimization
• Designing Neural Network Architectures using Reinforcement Learning
• Hierarchical Representations for Efficient Architecture Search
• MONAS: Multi-Objective Neural Architecture Search using Reinforcement Learning
• DeepArchitect: Automatically Designing and Training Deep Architectures
• Best Practices for Scientific Research on Neural Architecture Search
• SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers
• Exploring Randomly Wired Neural Networks for Image Recognition

Cited by in corpus (8)

• One Proxy Device Is Enough for Hardware-Aware Neural Architecture Search
• AutoHAS: Efficient Hyperparameter and Architecture Search
• Rethinking Co-design of Neural Architectures and Hardware Accelerators
• Primer: Searching for Efficient Transformers for Language Modeling
• CATE: Computation-aware Neural Architecture Encoding with Transformers
• Searching for Fast Model Families on Datacenter Accelerators
• Pi-NAS: Improving Neural Architecture Search by Reducing Supernet Training Consistency Shift
• Landmark Regularization: Ranking Guided Super-Net Training in Neural Architecture Search