Neural Architecture Generator Optimization

Abstract

Neural Architecture Search (NAS) was first proposed to achieve state-of-the-art performance through the discovery of new architecture patterns, without human intervention. An over-reliance on expert knowledge in the search space design has however led to increased performance (local optima) without significant architectural breakthroughs, thus preventing truly novel solutions from being reached. In this work we 1) are the first to investigate casting NAS as a problem of finding the optimal network generator and 2) we propose a new, hierarchical and graph-based search space capable of representing an extremely large variety of network types, yet only requiring few continuous hyper-parameters. This greatly reduces the dimensionality of the problem, enabling the effective use of Bayesian Optimisation as a search strategy. At the same time, we expand the range of valid architectures, motivating a multi-objective learning approach. We demonstrate the effectiveness of this strategy on six benchmark datasets and show that our search space generates extremely lightweight yet highly competitive models.

arXiv comment20 pages, 9 figures, neural architecture search, Thirty-fourth Conference on Neural Information Processing Systems

References in corpus (18)

• Practical Bayesian Optimization of Machine Learning Algorithms
• Neural Architecture Search with Reinforcement Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
• Neural Architecture Search: A Survey
• DARTS: Differentiable Architecture Search
• AutoAugment: Learning Augmentation Policies from Data
• Large-Scale Evolution of Image Classifiers
• Neural Architecture Optimization
• NAS-Bench-101: Towards Reproducible Neural Architecture Search
• Batch Bayesian Optimization via Local Penalization
• NAS evaluation is frustratingly hard
• NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
• Bayesian Optimization in AlphaGo
• Discovering Neural Wirings
• Asynchronous Batch Bayesian Optimisation with Improved Local Penalisation
• MANAS: Multi-Agent Neural Architecture Search
• Fast Neural Network Adaptation via Parameter Remapping and Architecture Search

Cited by in corpus (5)

• DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecture
• CATE: Computation-aware Neural Architecture Encoding with Transformers
• Pareto-Frontier-aware Neural Architecture Generation for Diverse Budgets
• Bag of Tricks for Neural Architecture Search
• Rapid Elastic Architecture Search under Specialized Classes and Resource Constraints