Rethinking Default Values: a Low Cost and Efficient Strategy to Define Hyperparameters

Abstract

Machine Learning (ML) algorithms have been increasingly applied to problems from several different areas. Despite their growing popularity, their predictive performance is usually affected by the values assigned to their hyperparameters (HPs). As consequence, researchers and practitioners face the challenge of how to set these values. Many users have limited knowledge about ML algorithms and the effect of their HP values and, therefore, do not take advantage of suitable settings. They usually define the HP values by trial and error, which is very subjective, not guaranteed to find good values and dependent on the user experience. Tuning techniques search for HP values able to maximize the predictive performance of induced models for a given dataset, but have the drawback of a high computational cost. Thus, practitioners use default values suggested by the algorithm developer or by tools implementing the algorithm. Although default values usually result in models with acceptable predictive performance, different implementations of the same algorithm can suggest distinct default values. To maintain a balance between tuning and using default values, we propose a strategy to generate new optimized default values. Our approach is grounded on a small set of optimized values able to obtain predictive performance values better than default settings provided by popular tools. After performing a large experiment and a careful analysis of the results, we concluded that our approach delivers better default values. Besides, it leads to competitive solutions when compared to tuned values, making it easier to use and having a lower cost. We also extracted simple rules to guide practitioners in deciding whether to use our new methodology or a HP tuning approach.

arXiv comment44 pages, 13 figures

References in corpus (7)

• Scikit-learn: Machine Learning in Python
• Practical Bayesian Optimization of Machine Learning Algorithms
• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
• OpenML: networked science in machine learning
• Tunability: Importance of Hyperparameters of Machine Learning Algorithms
• Importance of Tuning Hyperparameters of Machine Learning Algorithms
• A meta-learning recommender system for hyperparameter tuning: predicting when tuning improves SVM classifiers

Cited by in corpus (2)

• Constructing a meta-learner for unsupervised anomaly detection
• Meta-Learning for Symbolic Hyperparameter Defaults