Efficient Automatic CASH via Rising Bandits

Abstract

The Combined Algorithm Selection and Hyperparameter optimization (CASH) is one of the most fundamental problems in Automatic Machine Learning (AutoML). The existing Bayesian optimization (BO) based solutions turn the CASH problem into a Hyperparameter Optimization (HPO) problem by combining the hyperparameters of all machine learning (ML) algorithms, and use BO methods to solve it. As a result, these methods suffer from the low-efficiency problem due to the huge hyperparameter space in CASH. To alleviate this issue, we propose the alternating optimization framework, where the HPO problem for each ML algorithm and the algorithm selection problem are optimized alternately. In this framework, the BO methods are used to solve the HPO problem for each ML algorithm separately, incorporating a much smaller hyperparameter space for BO methods. Furthermore, we introduce Rising Bandits, a CASH-oriented Multi-Armed Bandits (MAB) variant, to model the algorithm selection in CASH. This framework can take the advantages of both BO in solving the HPO problem with a relatively small hyperparameter space and the MABs in accelerating the algorithm selection. Moreover, we further develop an efficient online algorithm to solve the Rising Bandits with provably theoretical guarantees. The extensive experiments on 30 OpenML datasets demonstrate the superiority of the proposed approach over the competitive baselines.

References in corpus (6)

• Practical Bayesian Optimization of Machine Learning Algorithms
• Automated Machine Learning: From Principles to Practices
• Non-stochastic Best Arm Identification and Hyperparameter Optimization
• BOHB: Robust and Efficient Hyperparameter Optimization at Scale
• Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets
• Multi-fidelity Bayesian Optimisation with Continuous Approximations

Cited by in corpus (6)

• OpenBox: A Generalized Black-box Optimization Service
• Efficient End-to-End AutoML via Scalable Search Space Decomposition
• VolcanoML: Speeding up End-to-End AutoML via Scalable Search Space Decomposition
• TransBO: Hyperparameter Optimization via Two-Phase Transfer Learning
• Transfer Learning based Search Space Design for Hyperparameter Tuning
• Facilitating Database Tuning with Hyper-Parameter Optimization: A Comprehensive Experimental Evaluation