Scalable Bayesian Optimization Using Deep Neural Networks

Abstract

Bayesian optimization is an effective methodology for the global optimization of functions with expensive evaluations. It relies on querying a distribution over functions defined by a relatively cheap surrogate model. An accurate model for this distribution over functions is critical to the effectiveness of the approach, and is typically fit using Gaussian processes (GPs). However, since GPs scale cubically with the number of observations, it has been challenging to handle objectives whose optimization requires many evaluations, and as such, massively parallelizing the optimization. In this work, we explore the use of neural networks as an alternative to GPs to model distributions over functions. We show that performing adaptive basis function regression with a neural network as the parametric form performs competitively with state-of-the-art GP-based approaches, but scales linearly with the number of data rather than cubically. This allows us to achieve a previously intractable degree of parallelism, which we apply to large scale hyperparameter optimization, rapidly finding competitive models on benchmark object recognition tasks using convolutional networks, and image caption generation using neural language models.

References in corpus (13)

• Auto-Encoding Variational Bayes
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Improving neural networks by preventing co-adaptation of feature detectors
• Practical Bayesian Optimization of Machine Learning Algorithms
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Recurrent Neural Network Regularization
• Gaussian Processes for Big Data
• Neural Variational Inference and Learning in Belief Networks
• Input Warping for Bayesian Optimization of Non-stationary Functions
• Portfolio Allocation for Bayesian Optimization
• Exponential Regret Bounds for Gaussian Process Bandits with Deterministic Observations
• Automated Machine Learning on Big Data using Stochastic Algorithm Tuning
• Anomaly Detection and Removal Using Non-Stationary Gaussian Processes

Cited by in corpus (166)

• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• AutoML: A Survey of the State-of-the-Art
• FractalNet: Ultra-Deep Neural Networks without Residuals
• Hyper-Parameter Optimization: A Review of Algorithms and Applications
• Application of Deep Convolutional Neural Networks for Detecting Extreme Weather in Climate Datasets
• Dealing with Integer-valued Variables in Bayesian Optimization with Gaussian Processes
• Dealing with Categorical and Integer-valued Variables in Bayesian Optimization with Gaussian Processes
• Deep Networks with Stochastic Depth
• CMA-ES for Hyperparameter Optimization of Deep Neural Networks
• Learning to Optimize Tensor Programs
• Spectral Representations for Convolutional Neural Networks
• Gryffin: An algorithm for Bayesian optimization of categorical variables informed by expert knowledge
• Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units
• Automated Machine Learning: State-of-The-Art and Open Challenges
• Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets
• An Ensemble of Epoch-wise Empirical Bayes for Few-shot Learning
• Deep Residual Networks with Exponential Linear Unit
• Do Deep Convolutional Nets Really Need to be Deep and Convolutional?
• Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness
• Predictive Entropy Search for Multi-objective Bayesian Optimization with Constraints
• Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS
• Uncertainty Quantification in Multivariable Regression for Material Property Prediction with Bayesian Neural Networks
• Physics discovery in nanoplasmonic systems via autonomous experiments in Scanning Transmission Electron Microscopy
• Probabilistic Matrix Factorization for Automated Machine Learning
• Iterative optimization of photonic crystal nanocavity designs by using deep neural networks
• Parallel Predictive Entropy Search for Batch Global Optimization of Expensive Objective Functions
• Metric Learning with Adaptive Density Discrimination
• MaskConnect: Connectivity Learning by Gradient Descent
• Robust Grasp Planning Over Uncertain Shape Completions
• Lets keep it simple, Using simple architectures to outperform deeper and more complex architectures
• Forward and Reverse Gradient-Based Hyperparameter Optimization
• Constrained Bayesian Optimization for Automatic Chemical Design
• ENTMOOT: A Framework for Optimization over Ensemble Tree Models
• Multi-Bias Non-linear Activation in Deep Neural Networks
• Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonfly
• Bayesian Hyperparameter Optimization for Ensemble Learning
• A Population-based Hybrid Approach to Hyperparameter Optimization for Neural Networks
• Hyp-RL : Hyperparameter Optimization by Reinforcement Learning
• Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?
• Quantifying Non-parametric Structure of High-redshift Galaxies with Deep Learning
• Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks
• Fast Bayesian Optimization of Needle-in-a-Haystack Problems using Zooming Memory-Based Initialization (ZoMBI)
• Finding Competitive Network Architectures Within a Day Using UCT
• Stochastic Gradient Line Bayesian Optimization for Efficient Noise-Robust Optimization of Parameterized Quantum Circuits
• Hyperparameter Ensembles for Robustness and Uncertainty Quantification
• 'In-Between' Uncertainty in Bayesian Neural Networks
• Multi-Objective Constrained Optimization for Energy Applications via Tree Ensembles
• Laplace Redux -- Effortless Bayesian Deep Learning
• A Stratified Analysis of Bayesian Optimization Methods
• MPC Controller Tuning using Bayesian Optimization Techniques
• AutoHAS: Efficient Hyperparameter and Architecture Search
• Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoML
• Multi-Fidelity Bayesian Optimization via Deep Neural Networks
• Ekya: Continuous Learning of Video Analytics Models on Edge Compute Servers
• Network of Experts for Large-Scale Image Categorization
• On Last-Layer Algorithms for Classification: Decoupling Representation from Uncertainty Estimation
• Noisy-Input Entropy Search for Efficient Robust Bayesian Optimization
• Neural Contextual Bandits with Deep Representation and Shallow Exploration
• How Powerful are Performance Predictors in Neural Architecture Search?
• Constrained Bayesian Optimization with Max-Value Entropy Search
• Unbounded Bayesian Optimization via Regularization
• Deep Kernel Learning
• GP-select: Accelerating EM using adaptive subspace preselection
• Parallel Architecture and Hyperparameter Search via Successive Halving and Classification
• Using a thousand optimization tasks to learn hyperparameter search strategies
• Mondrian Forests for Large-Scale Regression when Uncertainty Matters
• Hedging using reinforcement learning: Contextual $k$-Armed Bandit versus $Q$-learning
• Benchmarking the Neural Linear Model for Regression
• Bayesian Deep Learning via Subnetwork Inference
• Knowing The What But Not The Where in Bayesian Optimization
• A Novel Genetic Algorithm with Hierarchical Evaluation Strategy for Hyperparameter Optimisation of Graph Neural Networks
• A Gradient-based Bilevel Optimization Approach for Tuning Hyperparameters in Machine Learning
• Hyperboost: Hyperparameter Optimization by Gradient Boosting surrogate models
• Evolutionary Neural AutoML for Deep Learning
• Deep Q-Networks for Accelerating the Training of Deep Neural Networks
• Optimal Transport Kernels for Sequential and Parallel Neural Architecture Search
• Neural Network Architecture Optimization through Submodularity and Supermodularity
• Delta-STN: Efficient Bilevel Optimization for Neural Networks using Structured Response Jacobians
• Bayesian Adversarial Spheres: Bayesian Inference and Adversarial Examples in a Noiseless Setting
• PHOENICS: A universal deep Bayesian optimizer
• FLASH: Fast Bayesian Optimization for Data Analytic Pipelines
• Remote Sensor Design for Visual Recognition with Convolutional Neural Networks
• CATE: Computation-aware Neural Architecture Encoding with Transformers
• Evolutionary Architecture Search For Deep Multitask Networks
• Online Learning in Contextual Bandits using Gated Linear Networks
• Bayesian Optimization for Selecting Efficient Machine Learning Models
• Towards Assessing the Impact of Bayesian Optimization's Own Hyperparameters
• Deep Multi-Fidelity Active Learning of High-dimensional Outputs
• Faster Convergence in Deep-Predictive-Coding Networks to Learn Deeper Representations
• Lifelong Bayesian Optimization
• Solving Black-Box Optimization Challenge via Learning Search Space Partition for Local Bayesian Optimization
• Transferable Neural Processes for Hyperparameter Optimization
• Deep Neural Networks for Choice Analysis: Extracting Complete Economic Information for Interpretation
• Supervised COSMOS Autoencoder: Learning Beyond the Euclidean Loss!
• Neural Process for Black-Box Model Optimization Under Bayesian Framework
• An Asymptotically Optimal Multi-Armed Bandit Algorithm and Hyperparameter Optimization
• Scalable Thompson Sampling using Sparse Gaussian Process Models
• Beyond Point Estimate: Inferring Ensemble Prediction Variation from Neuron Activation Strength in Recommender Systems
• Neural Network Encapsulation
• The Nonlinearity Coefficient - A Practical Guide to Neural Architecture Design
• Unsupervised Learning using Pretrained CNN and Associative Memory Bank
• Scalable Training of Inference Networks for Gaussian-Process Models
• HyperSTAR: Task-Aware Hyperparameters for Deep Networks
• Genetic Neural Architecture Search for automatic assessment of human sperm images
• Adaptive Bayesian Linear Regression for Automated Machine Learning
• High Dimensional Restrictive Federated Model Selection with multi-objective Bayesian Optimization over shifted distributions
• Deep Bayesian Bandits: Exploring in Online Personalized Recommendations
• Adaptive Temporal-Difference Learning for Policy Evaluation with Per-State Uncertainty Estimates
• Practical Transfer Learning for Bayesian Optimization
• Sketch-Specific Data Augmentation for Freehand Sketch Recognition
• HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenML
• GradNets: Dynamic Interpolation Between Neural Architectures
• Neural Model-based Optimization with Right-Censored Observations
• Learning the Step-size Policy for the Limited-Memory Broyden-Fletcher-Goldfarb-Shanno Algorithm
• TS-UCB: Improving on Thompson Sampling With Little to No Additional Computation
• Data-Driven Offline Optimization For Architecting Hardware Accelerators
• Constrained Discrete Black-Box Optimization using Mixed-Integer Programming
• Regularized Evolutionary Population-Based Training
• Domain Knowledge Injection in Bayesian Search for New Materials
• Efficient Transfer Bayesian Optimization with Auxiliary Information
• Sparse Spectrum Gaussian Process for Bayesian Optimization
• Batch Multi-Fidelity Bayesian Optimization with Deep Auto-Regressive Networks
• Attentive Neural Processes and Batch Bayesian Optimization for Scalable Calibration of Physics-Informed Digital Twins
• RoMA: Robust Model Adaptation for Offline Model-based Optimization
• Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learning
• Online hyperparameter optimization by real-time recurrent learning
• BORE: Bayesian Optimization by Density-Ratio Estimation
• Hyperparameter Transfer Learning with Adaptive Complexity
• Convolutional Tables Ensemble: classification in microseconds
• Parameter-Based Value Functions
• Understanding Uncertainty in Bayesian Deep Learning
• Bayesian Linear Regression on Deep Representations
• Deep Neural Networks for Choice Analysis: Architectural Design with Alternative-Specific Utility Functions
• Discovering Diverse Athletic Jumping Strategies
• Funneled Bayesian Optimization for Design, Tuning and Control of Autonomous Systems
• Contact Map based Crystal Structure Prediction using Global Optimization
• Efficient Model Performance Estimation via Feature Histories
• A Hierarchical Approach to Scaling Batch Active Search Over Structured Data
• On Hyper-parameter Tuning for Stochastic Optimization Algorithms
• Gaussian Process Sampling and Optimization with Approximate Upper and Lower Bounds
• A Temporal Kernel Approach for Deep Learning with Continuous-time Information
• Explicit Gradient Learning
• Auto-CASH: Autonomous Classification Algorithm Selection with Deep Q-Network
• JUMBO: Scalable Multi-task Bayesian Optimization using Offline Data
• Posterior Temperature Optimization in Variational Inference for Inverse Problems
• Optimizing electrode positions in 2D Electrical Impedance Tomography using deep learning
• Scalable Bayesian Optimization with Sparse Gaussian Process Models
• PHS: A Toolbox for Parallel Hyperparameter Search
• Automatic hyperparameter selection in Autodock
• Collaborative Multidisciplinary Design Optimization with Neural Networks
• Adaptive Weight Decay for Deep Neural Networks
• Sonic: A Sampling-based Online Controller for Streaming Applications
• Approximate Bayesian Optimisation for Neural Networks
• Estimating air quality co-benefits of energy transition using machine learning
• Residual Overfit Method of Exploration
• Improving Hyperparameter Optimization by Planning Ahead
• Information Directed Reward Learning for Reinforcement Learning
• Black-Box Optimization of Object Detector Scales
• Cost-Efficient Online Hyperparameter Optimization
• Gradient-based Hyperparameter Optimization Over Long Horizons
• Explaining Inference Queries with Bayesian Optimization
• Adaptive Simulation-based Training of AI Decision-makers using Bayesian Optimization
• Training BatchNorm Only in Neural Architecture Search and Beyond
• Efficient Hyperparameter Optimization for Physics-based Character Animation
• Can a powerful neural network be a teacher for a weaker neural network?
• Differentiable Projection for Constrained Deep Learning