Practitioner Motives to Use Different Hyperparameter Optimization Methods

Abstract

Programmatic hyperparameter optimization (HPO) methods, such as Bayesian optimization and evolutionary algorithms, are highly sample-efficient in identifying optimal hyperparameter configurations for machine learning (ML) models. However, practitioners frequently use less efficient methods, such as grid search, which can lead to under-optimized models. We suspect this behavior is driven by a range of practitioner-specific motives. Practitioner motives, however, still need to be clarified to enhance user-centered development of HPO tools. To uncover practitioner motives to use different HPO methods, we conducted 20 semi-structured interviews and an online survey with 49 ML experts. By presenting main goals (e.g., increase ML model understanding) and contextual factors affecting practitioners' selection of HPO methods (e.g., available computer resources), this study offers a conceptual foundation to better understand why practitioners use different HPO methods, supporting development of more user-centered and context-adaptive HPO tools in automated ML.

References in corpus (14)

• On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization
• explAIner: A Visual Analytics Framework for Interactive and Explainable Machine Learning
• Trust in AutoML: Exploring Information Needs for Establishing Trust in Automated Machine Learning Systems
• Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overview
• ATMSeer: Increasing Transparency and Controllability in Automated Machine Learning
• SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization
• AutoDS: Towards Human-Centered Automation of Data Science
• Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonfly
• AutoML in The Wild: Obstacles, Workarounds, and Expectations
• AMLB: an AutoML Benchmark
• Incorporating Expert Prior in Bayesian Optimisation via Space Warping
• XAutoML: A Visual Analytics Tool for Understanding and Validating Automated Machine Learning