Freeze-Thaw Bayesian Optimization

Abstract

In this paper we develop a dynamic form of Bayesian optimization for machine learning models with the goal of rapidly finding good hyperparameter settings. Our method uses the partial information gained during the training of a machine learning model in order to decide whether to pause training and start a new model, or resume the training of a previously-considered model. We specifically tailor our method to machine learning problems by developing a novel positive-definite covariance kernel to capture a variety of training curves. Furthermore, we develop a Gaussian process prior that scales gracefully with additional temporal observations. Finally, we provide an information-theoretic framework to automate the decision process. Experiments on several common machine learning models show that our approach is extremely effective in practice.

References in corpus (3)

• Improving neural networks by preventing co-adaptation of feature detectors
• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
• Bayesian Optimization with Unknown Constraints

Cited by in corpus (55)

• Neural Architecture Search: A Survey
• Hyper-Parameter Optimization: A Review of Algorithms and Applications
• A Tutorial on Bayesian Optimization
• Non-stochastic Best Arm Identification and Hyperparameter Optimization
• Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets
• Hyperparameter optimization with approximate gradient
• Peephole: Predicting Network Performance Before Training
• Stochastic Hyperparameter Optimization through Hypernetworks
• Towards Green Automated Machine Learning: Status Quo and Future Directions
• Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonfly
• Bayesian Hyperparameter Optimization for Ensemble Learning
• Predicting Neural Network Accuracy from Weights
• Finding Competitive Network Architectures Within a Day Using UCT
• Model-based Asynchronous Hyperparameter and Neural Architecture Search
• Towards AutoML in the presence of Drift: first results
• A Probabilistic Machine Learning Approach to Scheduling Parallel Loops with Bayesian Optimization
• Practical Bayesian Optimization for Variable Cost Objectives
• Using a thousand optimization tasks to learn hyperparameter search strategies
• Fast Efficient Hyperparameter Tuning for Policy Gradients
• A Semi-Supervised Assessor of Neural Architectures
• Network Architecture Search for Domain Adaptation
• Sampled Training and Node Inheritance for Fast Evolutionary Neural Architecture Search
• From Federated Learning to Federated Neural Architecture Search: A Survey
• HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO
• Techniques for Automated Machine Learning
• Delta-STN: Efficient Bilevel Optimization for Neural Networks using Structured Response Jacobians
• TAPAS: Train-less Accuracy Predictor for Architecture Search
• Solving Compositional Reinforcement Learning Problems via Task Reduction
• AutoLRS: Automatic Learning-Rate Schedule by Bayesian Optimization on the Fly
• Automated Machine Learning with Monte-Carlo Tree Search
• HyperSTAR: Task-Aware Hyperparameters for Deep Networks
• Faster Improvement Rate Population Based Training
• DC-NAS: Divide-and-Conquer Neural Architecture Search
• A Novel Training Protocol for Performance Predictors of Evolutionary Neural Architecture Search Algorithms
• Bayesian Optimization for Iterative Learning
• Automatic model training under restrictive time constraints
• Probabilistic Rollouts for Learning Curve Extrapolation Across Hyperparameter Settings
• Weighting Is Worth the Wait: Bayesian Optimization with Importance Sampling
• BISTRO: Berkeley Integrated System for Transportation Optimization
• Open Loop Hyperparameter Optimization and Determinantal Point Processes
• Hyper-parameter Tuning under a Budget Constraint
• Non-smooth Bayesian Optimization in Tuning Problems
• Batch Multi-Fidelity Bayesian Optimization with Deep Auto-Regressive Networks
• Automatic Parallelization of Sequential Programs
• Adjoint-based trailing edge shape optimization of a transonic turbine vane using large eddy simulations
• apsis - Framework for Automated Optimization of Machine Learning Hyper Parameters
• Speed-Constrained Tuning for Statistical Machine Translation Using Bayesian Optimization
• Fast Hyperparameter Optimization of Deep Neural Networks via Ensembling Multiple Surrogates
• PEng4NN: An Accurate Performance Estimation Engine for Efficient Automated Neural Network Architecture Search
• Efficient Hyperparameter Optimization for Physics-based Character Animation
• Toward Optimal Run Racing: Application to Deep Learning Calibration
• Hyperparameter Transfer Learning through Surrogate Alignment for Efficient Deep Neural Network Training
• Preemptive Termination of Suggestions during Sequential Kriging Optimization of a Brain Activity Reconstruction Simulation
• Hypervolume-based Multi-objective Bayesian Optimization with Student-t Processes
• Cost-Efficient Online Hyperparameter Optimization