XNAS: Neural Architecture Search with Expert Advice

Abstract

This paper introduces a novel optimization method for differential neural architecture search, based on the theory of prediction with expert advice. Its optimization criterion is well fitted for an architecture-selection, i.e., it minimizes the regret incurred by a sub-optimal selection of operations. Unlike previous search relaxations, that require hard pruning of architectures, our method is designed to dynamically wipe out inferior architectures and enhance superior ones. It achieves an optimal worst-case regret bound and suggests the use of multiple learning-rates, based on the amount of information carried by the backward gradients. Experiments show that our algorithm achieves a strong performance over several image classification datasets. Specifically, it obtains an error rate of 1.6% for CIFAR-10, 24% for ImageNet under mobile settings, and achieves state-of-the-art results on three additional datasets.

References in corpus (10)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Neural Architecture Search with Reinforcement Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• On the Convergence of Adam and Beyond
• Random Erasing Data Augmentation
• Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation
• Exploring Randomly Wired Neural Networks for Image Recognition
• Probabilistic Neural Architecture Search
• The Freiburg Groceries Dataset
• sharpDARTS: Faster and More Accurate Differentiable Architecture Search

Cited by in corpus (5)

• NAS evaluation is frustratingly hard
• GOLD-NAS: Gradual, One-Level, Differentiable
• CIFAR-10 Image Classification Using Feature Ensembles
• Improving One-shot NAS by Suppressing the Posterior Fading
• ModuleNet: Knowledge-inherited Neural Architecture Search