Neural Architecture Search with Bayesian Optimisation and Optimal Transport
arXiv:1802.07191
Abstract
Bayesian Optimisation (BO) refers to a class of methods for global optimisation of a function which is only accessible via point evaluations. It is typically used in settings where is expensive to evaluate. A common use case for BO in machine learning is model selection, where it is not possible to analytically model the generalisation performance of a statistical model, and we resort to noisy and expensive training and validation procedures to choose the best model. Conventional BO methods have focused on Euclidean and categorical domains, which, in the context of model selection, only permits tuning scalar hyper-parameters of machine learning algorithms. However, with the surge of interest in deep learning, there is an increasing demand to tune neural network \emph{architectures}. In this work, we develop NASBOT, a Gaussian process based BO framework for neural architecture search. To accomplish this, we develop a distance metric in the space of neural network architectures which can be computed efficiently via an optimal transport program. This distance might be of independent interest to the deep learning community as it may find applications outside of BO. We demonstrate that NASBOT outperforms other alternatives for architecture search in several cross validation based model selection tasks on multi-layer perceptrons and convolutional neural networks.
References in corpus (15)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Practical Bayesian Optimization of Machine Learning Algorithms
- Neural Architecture Search with Reinforcement Learning
- A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
- Hierarchical Representations for Efficient Architecture Search
- Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP)
- Progressive Neural Architecture Search
- Contextual Decision Processes with Low Bellman Rank are PAC-Learnable
- DeepArchitect: Automatically Designing and Training Deep Architectures
- Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets
- AdaNet: Adaptive Structural Learning of Artificial Neural Networks
- Multi-fidelity Bayesian Optimisation with Continuous Approximations
- Raiders of the Lost Architecture: Kernels for Bayesian Optimization in Conditional Parameter Spaces
- Practical Block-wise Neural Network Architecture Generation
- The Multi-fidelity Multi-armed Bandit
Cited by in corpus (44)
- AutoML: A Survey of the State-of-the-Art
- A Survey on Neural Architecture Search
- Neuroevolution in Deep Neural Networks: Current Trends and Future Challenges
- Review: Deep Learning in Electron Microscopy
- NSGA-Net: Neural Architecture Search using Multi-Objective Genetic Algorithm
- Toward Extremely Lightweight Distracted Driver Recognition With Distillation-Based Neural Architecture Search and Knowledge Transfer
- Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
- Contrastive Self-supervised Neural Architecture Search
- Evaluating Efficient Performance Estimators of Neural Architectures
- AutoQ: Automated Kernel-Wise Neural Network Quantization
- Evolving Robust Neural Architectures to Defend from Adversarial Attacks
- Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
- Neural Ensemble Search for Uncertainty Estimation and Dataset Shift
- Mixed Precision Low-bit Quantization of Neural Network Language Models for Speech Recognition
- AutoPrivacy: Automated Layer-wise Parameter Selection for Secure Neural Network Inference
- Manas: Mining Software Repositories to Assist AutoML
- Parallel Architecture and Hyperparameter Search via Successive Halving and Classification
- Neural-BO: A Black-box Optimization Algorithm using Deep Neural Networks
- Neural network for determining an asteroid mineral composition from reflectance spectra
- GNAS: A Greedy Neural Architecture Search Method for Multi-Attribute Learning
- OutFin, a multi-device and multi-modal dataset for outdoor localization based on the fingerprinting approach
- Learnable Embedding Space for Efficient Neural Architecture Compression
- (433) Eros and (25143) Itokawa surface properties from reflectance spectra
- ProBO: Versatile Bayesian Optimization Using Any Probabilistic Programming Language
- Efficient Forward Architecture Search
- DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecture
- Fast Neural Architecture Construction using EnvelopeNets
- Rethinking the Number of Channels for the Convolutional Neural Network
- Neural Architecture Search by Estimation of Network Structure Distributions
- DARC: Differentiable ARchitecture Compression
- Latency-Aware Neural Architecture Search with Multi-Objective Bayesian Optimization
- Genetic Neural Architecture Search for automatic assessment of human sperm images
- Latent Domain Learning with Dynamic Residual Adapters
- Resource Constrained Neural Network Architecture Search: Will a Submodularity Assumption Help?
- Regret bounds for meta Bayesian optimization with an unknown Gaussian process prior
- Searching for Fast Model Families on Datacenter Accelerators
- Synthetic Petri Dish: A Novel Surrogate Model for Rapid Architecture Search
- Capacity allocation analysis of neural networks: A tool for principled architecture design
- Parameter Optimization using high-dimensional Bayesian Optimization
- NASIB: Neural Architecture Search withIn Budget
- iDARTS: Improving DARTS by Node Normalization and Decorrelation Discretization
- Topological Bayesian Optimization with Persistence Diagrams
- Sonic: A Sampling-based Online Controller for Streaming Applications
- Deep Bayesian Optimization on Attributed Graphs