Facilitating Database Tuning with Hyper-Parameter Optimization: A Comprehensive Experimental Evaluation
arXiv:2110.12654
Abstract
Recently, using automatic configuration tuning to improve the performance of modern database management systems (DBMSs) has attracted increasing interest from the database community. This is embodied with a number of systems featuring advanced tuning capabilities being developed. However, it remains a challenge to select the best solution for database configuration tuning, considering the large body of algorithm choices. In addition, beyond the applications on database systems, we could find more potential algorithms designed for configuration tuning. To this end, this paper provides a comprehensive evaluation of configuration tuning techniques from a broader perspective, hoping to better benefit the database community. In particular, we summarize three key modules of database configuration tuning systems and conduct extensive ablation studies using various challenging cases. Our evaluation demonstrates that the hyper-parameter optimization algorithms can be borrowed to further enhance the database configuration tuning. Moreover, we identify the best algorithm choices for different modules. Beyond the comprehensive evaluations, we offer an efficient and unified database configuration tuning benchmark via surrogates that reduces the evaluation cost to a minimum, allowing for extensive runs and analysis of new techniques.
References in corpus (16)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Continuous control with deep reinforcement learning
- Practical Bayesian Optimization of Machine Learning Algorithms
- On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice
- Large-Scale Evolution of Image Classifiers
- BayesOpt: A Bayesian Optimization Library for Nonlinear Optimization, Experimental Design and Bandits
- BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization
- Scalable Global Optimization via Local Bayesian Optimization
- Importance of Tuning Hyperparameters of Machine Learning Algorithms
- Batched Large-scale Bayesian Optimization in High-dimensional Spaces
- Bayesian Multi-Scale Optimistic Optimization
- Learning Search Space Partition for Black-box Optimization using Monte Carlo Tree Search
- Efficient Automatic CASH via Rising Bandits
- Hyperparameter Optimization and Boosting for Classifying Facial Expressions: How good can a "Null" Model be?
- Augmented Ensemble MCMC sampling in Factorial Hidden Markov Models
- MFES-HB: Efficient Hyperband with Multi-Fidelity Quality Measurements