XNAS: Neural Architecture Search with Expert Advice

Abstract

This paper introduces a novel optimization method for differential neural architecture search, based on the theory of prediction with expert advice. Its optimization criterion is well fitted for an architecture-selection, i.e., it minimizes the regret incurred by a sub-optimal selection of operations. Unlike previous search relaxations, that require hard pruning of architectures, our method is designed to dynamically wipe out inferior architectures and enhance superior ones. It achieves an optimal worst-case regret bound and suggests the use of multiple learning-rates, based on the amount of information carried by the backward gradients. Experiments show that our algorithm achieves a strong performance over several image classification datasets. Specifically, it obtains an error rate of 1.6% for CIFAR-10, 24% for ImageNet under mobile settings, and achieves state-of-the-art results on three additional datasets.

References in corpus (20)

• Adam: A Method for Stochastic Optimization
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Neural Architecture Search with Reinforcement Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• SGDR: Stochastic Gradient Descent with Warm Restarts
• On the Convergence of Adam and Beyond
• Random Erasing Data Augmentation
• AutoAugment: Learning Augmentation Policies from Data
• FractalNet: Ultra-Deep Neural Networks without Residuals
• Neural Architecture Optimization
• Random Search and Reproducibility for Neural Architecture Search
• Evaluating the Search Phase of Neural Architecture Search
• Learning Factored Representations in a Deep Mixture of Experts
• Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation
• Exploring Randomly Wired Neural Networks for Image Recognition
• Probabilistic Neural Architecture Search
• ASAP: Architecture Search, Anneal and Prune
• The Freiburg Groceries Dataset
• You Only Search Once: Single Shot Neural Architecture Search via Direct Sparse Optimization
• sharpDARTS: Faster and More Accurate Differentiable Architecture Search

Cited by in corpus (12)

• DARTS+: Improved Differentiable Architecture Search with Early Stopping
• NAS evaluation is frustratingly hard
• Review: Deep Learning in Electron Microscopy
• Stabilizing DARTS with Amended Gradient Estimation on Architectural Parameters
• GOLD-NAS: Gradual, One-Level, Differentiable
• CIFAR-10 Image Classification Using Feature Ensembles
• Hierarchical Neural Architecture Search via Operator Clustering
• Improving One-shot NAS by Suppressing the Posterior Fading
• Understanding Architectures Learnt by Cell-based Neural Architecture Search
• MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation
• ModuleNet: Knowledge-inherited Neural Architecture Search
• Large Scale Neural Architecture Search with Polyharmonic Splines