Practical Multi-fidelity Bayesian Optimization for Hyperparameter Tuning

Abstract

Bayesian optimization is popular for optimizing time-consuming black-box objectives. Nonetheless, for hyperparameter tuning in deep neural networks, the time required to evaluate the validation error for even a few hyperparameter settings remains a bottleneck. Multi-fidelity optimization promises relief using cheaper proxies to such objectives --- for example, validation error for a network trained using a subset of the training points or fewer iterations than required for convergence. We propose a highly flexible and practical approach to multi-fidelity Bayesian optimization, focused on efficiently optimizing hyperparameters for iteratively trained supervised learning models. We introduce a new acquisition function, the trace-aware knowledge-gradient, which efficiently leverages both multiple continuous fidelity controls and trace observations --- values of the objective at a sequence of fidelities, available when varying fidelity using training iterations. We provide a provably convergent method for optimizing our acquisition function and show it outperforms state-of-the-art alternatives for hyperparameter tuning of deep neural networks and large-scale kernel learning.

Cited by in corpus (7)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Fast and Accurate Road Crack Detection Based on Adaptive Cost-Sensitive Loss Function
• Towards Green Automated Machine Learning: Status Quo and Future Directions
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
• Leveraging Trust for Joint Multi-Objective and Multi-Fidelity Optimization
• Problem-fluent models for complex decision-making in autonomous materials research
• Bayesian optimization for modular black-box systems with switching costs