Tunability: Importance of Hyperparameters of Machine Learning Algorithms

Abstract

Modern supervised machine learning algorithms involve hyperparameters that have to be set before running them. Options for setting hyperparameters are default values from the software package, manual configuration by the user or configuring them for optimal predictive performance by a tuning procedure. The goal of this paper is two-fold. Firstly, we formalize the problem of tuning from a statistical point of view, define data-based defaults and suggest general measures quantifying the tunability of hyperparameters of algorithms. Secondly, we conduct a large-scale benchmarking study based on 38 datasets from the OpenML platform and six common machine learning algorithms. We apply our measures to assess the tunability of their parameters. Our results yield default values for hyperparameters and enable users to decide whether it is worth conducting a possibly time consuming tuning strategy, to focus on the most important hyperparameters and to chose adequate hyperparameter spaces for tuning.

arXiv comment22 pages, 10 tables, 8 figures

References in corpus (3)

• mlrMBO: A Modular Framework for Model-Based Optimization of Expensive Black-Box Functions
• OpenML: An R Package to Connect to the Machine Learning Platform OpenML
• OpenML Benchmarking Suites

Cited by in corpus (21)

• Hyperparameters and Tuning Strategies for Random Forest
• Performance evaluation and hyperparameter tuning of statistical and machine-learning models using spatial data
• Importance of Tuning Hyperparameters of Machine Learning Algorithms
• Review: Deep Learning in Electron Microscopy
• Assuring the Machine Learning Lifecycle: Desiderata, Methods, and Challenges
• Automated Evolutionary Approach for the Design of Composite Machine Learning Pipelines
• HyperTendril: Visual Analytics for User-Driven Hyperparameter Optimization of Deep Neural Networks
• VisEvol: Visual Analytics to Support Hyperparameter Search through Evolutionary Optimization
• Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoML
• Maintainability Challenges in ML: A Systematic Literature Review
• Two new feature selection methods based on learn-heuristic techniques for breast cancer prediction: A comprehensive analysis
• Rethinking Default Values: a Low Cost and Efficient Strategy to Define Hyperparameters
• Optimization of Image Processing Algorithms for Character Recognition in Cultural Typewritten Documents
• What Makes Forest-Based Heterogeneous Treatment Effect Estimators Work?
• TensorSocket: Shared Data Loading for Deep Learning Training
• High Dimensional Restrictive Federated Model Selection with multi-objective Bayesian Optimization over shifted distributions
• Boosting Resource-Constrained Federated Learning Systems with Guessed Updates
• Cluster optical depth and pairwise velocity estimation using machine learning
• What augmentations are sensitive to hyper-parameters and why?
• Optimizing Emotion Recognition with Wearable Sensor Data: Unveiling Patterns in Body Movements and Heart Rate through Random Forest Hyperparameter Tuning
• Personalised dynamic super learning: an application in predicting hemodiafiltration convection volumes