Hyperparameter Importance Across Datasets

Abstract

With the advent of automated machine learning, automated hyperparameter optimization methods are by now routinely used in data mining. However, this progress is not yet matched by equal progress on automatic analyses that yield information beyond performance-optimizing hyperparameter settings. In this work, we aim to answer the following two questions: Given an algorithm, what are generally its most important hyperparameters, and what are typically good values for these? We present methodology and a framework to answer these questions based on meta-learning across many datasets. We apply this methodology using the experimental meta-data available on OpenML to determine the most important hyperparameters of support vector machines, random forests and Adaboost, and to infer priors for all their hyperparameters. The results, obtained fully automatically, provide a quantitative basis to focus efforts in both manual algorithm design and in automated hyperparameter optimization. The conducted experiments confirm that the hyperparameters selected by the proposed method are indeed the most important ones and that the obtained priors also lead to statistically significant improvements in hyperparameter optimization.

\c{opyright} 2018. Copyright is held by the owner/author(s). Publication rights licensed to ACM. This is the author's version of the work. It is posted here for your personal use, not for redistribution. The definitive Version of Record was published in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

References in corpus (1)

• OpenML: networked science in machine learning

Cited by in corpus (12)

• Hyperparameters and Tuning Strategies for Random Forest
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• HyperTendril: Visual Analytics for User-Driven Hyperparameter Optimization of Deep Neural Networks
• Efficient End-to-End AutoML via Scalable Search Space Decomposition
• VisEvol: Visual Analytics to Support Hyperparameter Search through Evolutionary Optimization
• Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoML
• Incorporating Expert Prior in Bayesian Optimisation via Space Warping
• VolcanoML: Speeding up End-to-End AutoML via Scalable Search Space Decomposition
• High-dimensional Automated Radiation Therapy Treatment Planning via Bayesian Optimization
• Hyperparameter Importance of Quantum Neural Networks Across Small Datasets
• Regularized boosting with an increasing coefficient magnitude stop criterion as meta-learner in hyperparameter optimization stacking ensemble
• On the Tunability of Random Survival Forests Model for Predictive Maintenance