One-Shot Neural Architecture Search via Self-Evaluated Template Network

Abstract

Neural architecture search (NAS) aims to automate the search procedure of architecture instead of manual design. Even if recent NAS approaches finish the search within days, lengthy training is still required for a specific architecture candidate to get the parameters for its accurate evaluation. Recently one-shot NAS methods are proposed to largely squeeze the tedious training process by sharing parameters across candidates. In this way, the parameters for each candidate can be directly extracted from the shared parameters instead of training them from scratch. However, they have no sense of which candidate will perform better until evaluation so that the candidates to evaluate are randomly sampled and the top-1 candidate is considered the best. In this paper, we propose a Self-Evaluated Template Network (SETN) to improve the quality of the architecture candidates for evaluation so that it is more likely to cover competitive candidates. SETN consists of two components: (1) an evaluator, which learns to indicate the probability of each individual architecture being likely to have a lower validation loss. The candidates for evaluation can thus be selectively sampled according to this evaluator. (2) a template network, which shares parameters among all candidates to amortize the training cost of generated candidates. In experiments, the architecture found by SETN achieves state-of-the-art performance on CIFAR and ImageNet benchmarks within comparable computation costs. Code is publicly available on GitHub: https://github.com/D-X-Y/AutoDL-Projects.

arXiv commentMinor modifications to the ICCV 2019 camera-ready version (add code link)

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Neural Architecture Search with Reinforcement Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• DARTS: Differentiable Architecture Search
• Large-Scale Evolution of Image Classifiers
• Neural Architecture Optimization
• Designing Neural Network Architectures using Reinforcement Learning
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• Efficient Architecture Search by Network Transformation
• NAS-Bench-101: Towards Reproducible Neural Architecture Search
• Neural Optimizer Search with Reinforcement Learning
• Network Pruning via Transformable Architecture Search
• Graph HyperNetworks for Neural Architecture Search
• Densely Connected Search Space for More Flexible Neural Architecture Search

Cited by in corpus (45)

• AutoML: A Survey of the State-of-the-Art
• A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
• NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size
• NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search
• EPE-NAS: Efficient Performance Estimation Without Training for Neural Architecture Search
• Neural Architecture Search without Training
• Noisy Differentiable Architecture Search
• Evaluating Efficient Performance Estimators of Neural Architectures
• DARTS-: Robustly Stepping out of Performance Collapse Without Indicators
• Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• PRE-NAS: Predictor-assisted Evolutionary Neural Architecture Search
• Angle-based Search Space Shrinking for Neural Architecture Search
• AdversarialNAS: Adversarial Neural Architecture Search for GANs
• Less is More: Proxy Datasets in NAS approaches
• From Federated Learning to Federated Neural Architecture Search: A Survey
• Cyclic Differentiable Architecture Search
• MTL-NAS: Task-Agnostic Neural Architecture Search towards General-Purpose Multi-Task Learning
• Few-shot Neural Architecture Search
• NAS-Navigator: Visual Steering for Explainable One-Shot Deep Neural Network Synthesis
• TF-NAS: Rethinking Three Search Freedoms of Latency-Constrained Differentiable Neural Architecture Search
• RBFleX-NAS: Training-Free Neural Architecture Search Using Radial Basis Function Kernel and Hyperparameter Detection
• Automated Dominative Subspace Mining for Efficient Neural Architecture Search
• Rapid Neural Architecture Search by Learning to Generate Graphs from Datasets
• On the Privacy Risks of Cell-Based NAS Architectures
• HourNAS: Extremely Fast Neural Architecture Search Through an Hourglass Lens
• LT-Net: Label Transfer by Learning Reversible Voxel-wise Correspondence for One-shot Medical Image Segmentation
• A Deep Neural Networks ensemble workflow from hyperparameter search to inference leveraging GPU clusters
• DOTS: Decoupling Operation and Topology in Differentiable Architecture Search
• FNA++: Fast Network Adaptation via Parameter Remapping and Architecture Search
• Generic Neural Architecture Search via Regression
• Single-level Optimization For Differential Architecture Search
• Robustifying DARTS by Eliminating Information Bypass Leakage via Explicit Sparse Regularization
• Evolutionary Neural Cascade Search across Supernetworks
• Mixed Variable Bayesian Optimization with Frequency Modulated Kernels
• BaLeNAS: Differentiable Architecture Search via the Bayesian Learning Rule
• Neural Parameter Allocation Search
• Differentiable Feature Aggregation Search for Knowledge Distillation
• Memory-Efficient Hierarchical Neural Architecture Search for Image Restoration
• Guided Evolution for Neural Architecture Search
• Training BatchNorm Only in Neural Architecture Search and Beyond
• Connection Sensitivity Matters for Training-free DARTS: From Architecture-Level Scoring to Operation-Level Sensitivity Analysis
• Searching by Generating: Flexible and Efficient One-Shot NAS with Architecture Generator
• Reconsidering CO2 emissions from Computer Vision
• Single-DARTS: Towards Stable Architecture Search